2026-03-20T17:57:43.322 INFO:root:teuthology version: 1.2.4.dev6+g1c580df7a 2026-03-20T17:57:43.327 DEBUG:teuthology.report:Pushing job info to http://localhost:8080 2026-03-20T17:57:43.346 INFO:teuthology.run:Config: archive_path: /archive/kyr-2026-03-20_17:25:16-rgw-tentacle-none-default-vps/2230 branch: tentacle description: rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/amqp/{0-install centos_latest test_amqp} valgrind} email: null first_in_suite: false flavor: default job_id: '2230' last_in_suite: false machine_type: vps name: kyr-2026-03-20_17:25:16-rgw-tentacle-none-default-vps no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: tentacle ansible.cephlab: branch: main repo: https://github.com/kshtsk/ceph-cm-ansible.git skip_tags: nagios,monitoring-scripts,hostname,pubkeys,zap,sudoers,kerberos,ntp-client,resolvconf,cpan,nfs vars: logical_volumes: lv_1: scratch_dev: true size: 25%VG vg: vg_nvme lv_2: scratch_dev: true size: 25%VG vg: vg_nvme lv_3: scratch_dev: true size: 25%VG vg: vg_nvme lv_4: scratch_dev: true size: 25%VG vg: vg_nvme timezone: UTC volume_groups: vg_nvme: pvs: /dev/vdb,/dev/vdc,/dev/vdd,/dev/vde ceph: conf: client: debug rgw: 20 debug rgw lifecycle: 20 debug rgw notification: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= setgroup: ceph setuser: ceph global: osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd mclock iops capacity threshold hdd: 49000 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 70f8415b300f041766fa27faf7d5472699e32388 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs cephadm: cephadm_binary_url: https://download.ceph.com/rpm-20.2.0/el9/noarch/cephadm install: ceph: flavor: default sha1: 70f8415b300f041766fa27faf7d5472699e32388 extra_system_packages: deb: - python3-jmespath - python3-xmltodict - s3cmd rpm: - bzip2 - perl-Test-Harness - python3-jmespath - python3-xmltodict - s3cmd rgw: client.0: valgrind: - --tool=memcheck - --max-threads=1024 client.1: valgrind: - --tool=memcheck - --max-threads=1024 frontend: beast realm: MyRealm storage classes: FROZEN: null LUKEWARM: null zone: MyZone zonegroup: MyZoneGroup thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: tt-tentacle sha1: c04fae7bd18063e6f8f9e1cf40c23f810c1f6ca4 owner: kyr priority: 1000 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 5336 sha1: 70f8415b300f041766fa27faf7d5472699e32388 sleep_before_teardown: 0 suite: rgw suite_branch: tt-tentacle suite_path: /home/teuthos/src/github.com_kshtsk_ceph_c04fae7bd18063e6f8f9e1cf40c23f810c1f6ca4/qa suite_relpath: qa suite_repo: https://github.com/kshtsk/ceph.git suite_sha1: c04fae7bd18063e6f8f9e1cf40c23f810c1f6ca4 targets: vm01.local: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPM2K9GoxhY4gNP+c5/VZH2o9RRCjh6T6aeNvsve1YRthc4ggz38D3QZ8AGNKxSajVe0BXWgGrDQaoDSTLWhEKk= vm03.local: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCubnQzHt2zSKsan2cmD2y4ZJofSCUgPlfmIiloKSdwKBwzFDR7uruYPmhW3I5f5xx8/aHnYo4KtqyanMsoJvp4= tasks: - install: null - ceph: null - openssl_keys: null - rgw: client.0: null - rabbitmq: client.0: null - notification-tests: client.0: extra_attr: - amqp_test rgw_server: client.0 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: clyso-debian-13 teuthology_repo: https://github.com/clyso/teuthology teuthology_sha1: 1c580df7a9c7c2aadc272da296344fd99f27c444 timestamp: 2026-03-20_17:25:16 tube: vps user: kyr verbose: false worker_log: /home/teuthos/.teuthology/dispatcher/dispatcher.vps.4188345 2026-03-20T17:57:43.346 INFO:teuthology.run:suite_path is set to /home/teuthos/src/github.com_kshtsk_ceph_c04fae7bd18063e6f8f9e1cf40c23f810c1f6ca4/qa; will attempt to use it 2026-03-20T17:57:43.347 INFO:teuthology.run:Found tasks at /home/teuthos/src/github.com_kshtsk_ceph_c04fae7bd18063e6f8f9e1cf40c23f810c1f6ca4/qa/tasks 2026-03-20T17:57:43.347 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-03-20T17:57:43.347 INFO:teuthology.task.internal:Checking packages... 2026-03-20T17:57:43.347 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '70f8415b300f041766fa27faf7d5472699e32388' 2026-03-20T17:57:43.347 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-03-20T17:57:43.347 INFO:teuthology.packaging:ref: None 2026-03-20T17:57:43.347 INFO:teuthology.packaging:tag: None 2026-03-20T17:57:43.347 INFO:teuthology.packaging:branch: tentacle 2026-03-20T17:57:43.347 INFO:teuthology.packaging:sha1: 70f8415b300f041766fa27faf7d5472699e32388 2026-03-20T17:57:43.347 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=tentacle 2026-03-20T17:57:44.033 INFO:teuthology.task.internal:Found packages for ceph version 20.2.0-721.g5bb32787 2026-03-20T17:57:44.034 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-03-20T17:57:44.034 INFO:teuthology.task.internal:no buildpackages task found 2026-03-20T17:57:44.034 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-03-20T17:57:44.035 INFO:teuthology.task.internal:Saving configuration 2026-03-20T17:57:44.040 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-03-20T17:57:44.041 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-03-20T17:57:44.047 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'vm01.local', 'description': '/archive/kyr-2026-03-20_17:25:16-rgw-tentacle-none-default-vps/2230', 'up': True, 'machine_type': 'vps', 'is_vm': True, 'vm_host': {'name': 'localhost', 'description': None, 'up': True, 'machine_type': 'libvirt', 'is_vm': False, 'vm_host': None, 'os_type': None, 'os_version': None, 'arch': None, 'locked': True, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': None}, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-03-20 17:56:35.360257', 'locked_by': 'kyr', 'mac_address': '52:55:00:00:00:01', 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPM2K9GoxhY4gNP+c5/VZH2o9RRCjh6T6aeNvsve1YRthc4ggz38D3QZ8AGNKxSajVe0BXWgGrDQaoDSTLWhEKk='} 2026-03-20T17:57:44.053 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'vm03.local', 'description': '/archive/kyr-2026-03-20_17:25:16-rgw-tentacle-none-default-vps/2230', 'up': True, 'machine_type': 'vps', 'is_vm': True, 'vm_host': {'name': 'localhost', 'description': None, 'up': True, 'machine_type': 'libvirt', 'is_vm': False, 'vm_host': None, 'os_type': None, 'os_version': None, 'arch': None, 'locked': True, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': None}, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-03-20 17:56:35.360689', 'locked_by': 'kyr', 'mac_address': '52:55:00:00:00:03', 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCubnQzHt2zSKsan2cmD2y4ZJofSCUgPlfmIiloKSdwKBwzFDR7uruYPmhW3I5f5xx8/aHnYo4KtqyanMsoJvp4='} 2026-03-20T17:57:44.053 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-03-20T17:57:44.053 INFO:teuthology.task.internal:roles: ubuntu@vm01.local - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2026-03-20T17:57:44.053 INFO:teuthology.task.internal:roles: ubuntu@vm03.local - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2026-03-20T17:57:44.053 INFO:teuthology.run_tasks:Running task console_log... 2026-03-20T17:57:44.059 DEBUG:teuthology.task.console_log:vm01 does not support IPMI; excluding 2026-03-20T17:57:44.065 DEBUG:teuthology.task.console_log:vm03 does not support IPMI; excluding 2026-03-20T17:57:44.065 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe7946184c0>, signals=[15]) 2026-03-20T17:57:44.065 INFO:teuthology.run_tasks:Running task internal.connect... 2026-03-20T17:57:44.066 INFO:teuthology.task.internal:Opening connections... 2026-03-20T17:57:44.066 DEBUG:teuthology.task.internal:connecting to ubuntu@vm01.local 2026-03-20T17:57:44.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'vm01.local', 'username': 'ubuntu', 'timeout': 60} 2026-03-20T17:57:44.124 DEBUG:teuthology.task.internal:connecting to ubuntu@vm03.local 2026-03-20T17:57:44.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'vm03.local', 'username': 'ubuntu', 'timeout': 60} 2026-03-20T17:57:44.181 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-03-20T17:57:44.182 DEBUG:teuthology.orchestra.run.vm01:> uname -m 2026-03-20T17:57:44.217 INFO:teuthology.orchestra.run.vm01.stdout:x86_64 2026-03-20T17:57:44.217 DEBUG:teuthology.orchestra.run.vm01:> cat /etc/os-release 2026-03-20T17:57:44.271 INFO:teuthology.orchestra.run.vm01.stdout:NAME="CentOS Stream" 2026-03-20T17:57:44.271 INFO:teuthology.orchestra.run.vm01.stdout:VERSION="9" 2026-03-20T17:57:44.271 INFO:teuthology.orchestra.run.vm01.stdout:ID="centos" 2026-03-20T17:57:44.272 INFO:teuthology.orchestra.run.vm01.stdout:ID_LIKE="rhel fedora" 2026-03-20T17:57:44.272 INFO:teuthology.orchestra.run.vm01.stdout:VERSION_ID="9" 2026-03-20T17:57:44.272 INFO:teuthology.orchestra.run.vm01.stdout:PLATFORM_ID="platform:el9" 2026-03-20T17:57:44.272 INFO:teuthology.orchestra.run.vm01.stdout:PRETTY_NAME="CentOS Stream 9" 2026-03-20T17:57:44.272 INFO:teuthology.orchestra.run.vm01.stdout:ANSI_COLOR="0;31" 2026-03-20T17:57:44.272 INFO:teuthology.orchestra.run.vm01.stdout:LOGO="fedora-logo-icon" 2026-03-20T17:57:44.272 INFO:teuthology.orchestra.run.vm01.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-03-20T17:57:44.272 INFO:teuthology.orchestra.run.vm01.stdout:HOME_URL="https://centos.org/" 2026-03-20T17:57:44.272 INFO:teuthology.orchestra.run.vm01.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-03-20T17:57:44.272 INFO:teuthology.orchestra.run.vm01.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-03-20T17:57:44.272 INFO:teuthology.orchestra.run.vm01.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-03-20T17:57:44.272 INFO:teuthology.lock.ops:Updating vm01.local on lock server 2026-03-20T17:57:44.276 DEBUG:teuthology.orchestra.run.vm03:> uname -m 2026-03-20T17:57:44.289 INFO:teuthology.orchestra.run.vm03.stdout:x86_64 2026-03-20T17:57:44.289 DEBUG:teuthology.orchestra.run.vm03:> cat /etc/os-release 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:NAME="CentOS Stream" 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:VERSION="9" 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:ID="centos" 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:ID_LIKE="rhel fedora" 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:VERSION_ID="9" 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:PLATFORM_ID="platform:el9" 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:PRETTY_NAME="CentOS Stream 9" 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:ANSI_COLOR="0;31" 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:LOGO="fedora-logo-icon" 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:HOME_URL="https://centos.org/" 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-03-20T17:57:44.342 INFO:teuthology.orchestra.run.vm03.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-03-20T17:57:44.343 INFO:teuthology.orchestra.run.vm03.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-03-20T17:57:44.343 INFO:teuthology.lock.ops:Updating vm03.local on lock server 2026-03-20T17:57:44.346 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-03-20T17:57:44.348 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-03-20T17:57:44.349 INFO:teuthology.task.internal:Checking for old test directory... 2026-03-20T17:57:44.349 DEBUG:teuthology.orchestra.run.vm01:> test '!' -e /home/ubuntu/cephtest 2026-03-20T17:57:44.350 DEBUG:teuthology.orchestra.run.vm03:> test '!' -e /home/ubuntu/cephtest 2026-03-20T17:57:44.396 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-03-20T17:57:44.397 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-03-20T17:57:44.397 DEBUG:teuthology.orchestra.run.vm01:> test -z $(ls -A /var/lib/ceph) 2026-03-20T17:57:44.404 DEBUG:teuthology.orchestra.run.vm03:> test -z $(ls -A /var/lib/ceph) 2026-03-20T17:57:44.417 INFO:teuthology.orchestra.run.vm01.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-03-20T17:57:44.450 INFO:teuthology.orchestra.run.vm03.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-03-20T17:57:44.450 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-03-20T17:57:44.457 DEBUG:teuthology.orchestra.run.vm01:> test -e /ceph-qa-ready 2026-03-20T17:57:44.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-03-20T17:57:44.666 DEBUG:teuthology.orchestra.run.vm03:> test -e /ceph-qa-ready 2026-03-20T17:57:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-03-20T17:57:44.887 INFO:teuthology.run_tasks:Running task internal.base... 2026-03-20T17:57:44.889 INFO:teuthology.task.internal:Creating test directory... 2026-03-20T17:57:44.889 DEBUG:teuthology.orchestra.run.vm01:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-03-20T17:57:44.891 DEBUG:teuthology.orchestra.run.vm03:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-03-20T17:57:44.906 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-03-20T17:57:44.908 INFO:teuthology.run_tasks:Running task internal.archive... 2026-03-20T17:57:44.909 INFO:teuthology.task.internal:Creating archive directory... 2026-03-20T17:57:44.909 DEBUG:teuthology.orchestra.run.vm01:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-03-20T17:57:44.948 DEBUG:teuthology.orchestra.run.vm03:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-03-20T17:57:44.964 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-03-20T17:57:44.966 INFO:teuthology.task.internal:Enabling coredump saving... 2026-03-20T17:57:44.966 DEBUG:teuthology.orchestra.run.vm01:> test -f /run/.containerenv -o -f /.dockerenv 2026-03-20T17:57:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-03-20T17:57:45.017 DEBUG:teuthology.orchestra.run.vm03:> test -f /run/.containerenv -o -f /.dockerenv 2026-03-20T17:57:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-03-20T17:57:45.029 DEBUG:teuthology.orchestra.run.vm01:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-03-20T17:57:45.058 DEBUG:teuthology.orchestra.run.vm03:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-03-20T17:57:45.081 INFO:teuthology.orchestra.run.vm01.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-03-20T17:57:45.091 INFO:teuthology.orchestra.run.vm01.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-03-20T17:57:45.092 INFO:teuthology.orchestra.run.vm03.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-03-20T17:57:45.101 INFO:teuthology.orchestra.run.vm03.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-03-20T17:57:45.103 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-03-20T17:57:45.104 INFO:teuthology.task.internal:Configuring sudo... 2026-03-20T17:57:45.104 DEBUG:teuthology.orchestra.run.vm01:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-03-20T17:57:45.134 DEBUG:teuthology.orchestra.run.vm03:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-03-20T17:57:45.168 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-03-20T17:57:45.170 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-03-20T17:57:45.170 DEBUG:teuthology.orchestra.run.vm01:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-03-20T17:57:45.196 DEBUG:teuthology.orchestra.run.vm03:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-03-20T17:57:45.221 DEBUG:teuthology.orchestra.run.vm01:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-03-20T17:57:45.269 DEBUG:teuthology.orchestra.run.vm01:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-03-20T17:57:45.324 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T17:57:45.324 DEBUG:teuthology.orchestra.run.vm01:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-03-20T17:57:45.383 DEBUG:teuthology.orchestra.run.vm03:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-03-20T17:57:45.405 DEBUG:teuthology.orchestra.run.vm03:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-03-20T17:57:45.459 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T17:57:45.459 DEBUG:teuthology.orchestra.run.vm03:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-03-20T17:57:45.516 DEBUG:teuthology.orchestra.run.vm01:> sudo service rsyslog restart 2026-03-20T17:57:45.518 DEBUG:teuthology.orchestra.run.vm03:> sudo service rsyslog restart 2026-03-20T17:57:45.543 INFO:teuthology.orchestra.run.vm01.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-03-20T17:57:45.581 INFO:teuthology.orchestra.run.vm03.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-03-20T17:57:46.044 INFO:teuthology.run_tasks:Running task internal.timer... 2026-03-20T17:57:46.046 INFO:teuthology.task.internal:Starting timer... 2026-03-20T17:57:46.046 INFO:teuthology.run_tasks:Running task pcp... 2026-03-20T17:57:46.049 INFO:teuthology.run_tasks:Running task selinux... 2026-03-20T17:57:46.051 INFO:teuthology.task.selinux:Excluding vm01: VMs are not yet supported 2026-03-20T17:57:46.051 INFO:teuthology.task.selinux:Excluding vm03: VMs are not yet supported 2026-03-20T17:57:46.051 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-03-20T17:57:46.051 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-03-20T17:57:46.051 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-03-20T17:57:46.051 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-03-20T17:57:46.052 DEBUG:teuthology.task:Applying overrides for task ansible.cephlab: {'branch': 'main', 'repo': 'https://github.com/kshtsk/ceph-cm-ansible.git', 'skip_tags': 'nagios,monitoring-scripts,hostname,pubkeys,zap,sudoers,kerberos,ntp-client,resolvconf,cpan,nfs', 'vars': {'logical_volumes': {'lv_1': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_2': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_3': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_4': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}}, 'timezone': 'UTC', 'volume_groups': {'vg_nvme': {'pvs': '/dev/vdb,/dev/vdc,/dev/vdd,/dev/vde'}}}} 2026-03-20T17:57:46.053 DEBUG:teuthology.repo_utils:Setting repo remote to https://github.com/kshtsk/ceph-cm-ansible.git 2026-03-20T17:57:46.054 INFO:teuthology.repo_utils:Fetching github.com_kshtsk_ceph-cm-ansible_main from origin 2026-03-20T17:57:46.515 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthos/src/github.com_kshtsk_ceph-cm-ansible_main to origin/main 2026-03-20T17:57:46.520 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-03-20T17:57:46.520 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu", "logical_volumes": {"lv_1": {"scratch_dev": true, "size": "25%VG", "vg": "vg_nvme"}, "lv_2": {"scratch_dev": true, "size": "25%VG", "vg": "vg_nvme"}, "lv_3": {"scratch_dev": true, "size": "25%VG", "vg": "vg_nvme"}, "lv_4": {"scratch_dev": true, "size": "25%VG", "vg": "vg_nvme"}}, "timezone": "UTC", "volume_groups": {"vg_nvme": {"pvs": "/dev/vdb,/dev/vdc,/dev/vdd,/dev/vde"}}}' -i /tmp/teuth_ansible_inventorym7db9sc3 --limit vm01.local,vm03.local /home/teuthos/src/github.com_kshtsk_ceph-cm-ansible_main/cephlab.yml --skip-tags nagios,monitoring-scripts,hostname,pubkeys,zap,sudoers,kerberos,ntp-client,resolvconf,cpan,nfs 2026-03-20T17:59:17.937 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@vm01.local'), Remote(name='ubuntu@vm03.local')] 2026-03-20T17:59:17.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@vm01.local' 2026-03-20T17:59:17.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'vm01.local', 'username': 'ubuntu', 'timeout': 60} 2026-03-20T17:59:17.998 DEBUG:teuthology.orchestra.run.vm01:> true 2026-03-20T17:59:18.072 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@vm01.local' 2026-03-20T17:59:18.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@vm03.local' 2026-03-20T17:59:18.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'vm03.local', 'username': 'ubuntu', 'timeout': 60} 2026-03-20T17:59:18.131 DEBUG:teuthology.orchestra.run.vm03:> true 2026-03-20T17:59:18.204 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@vm03.local' 2026-03-20T17:59:18.204 INFO:teuthology.run_tasks:Running task clock... 2026-03-20T17:59:18.207 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-03-20T17:59:18.207 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-03-20T17:59:18.207 DEBUG:teuthology.orchestra.run.vm01:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-03-20T17:59:18.209 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-03-20T17:59:18.209 DEBUG:teuthology.orchestra.run.vm03:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-03-20T17:59:18.239 INFO:teuthology.orchestra.run.vm01.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-03-20T17:59:18.254 INFO:teuthology.orchestra.run.vm01.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-03-20T17:59:18.278 INFO:teuthology.orchestra.run.vm03.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-03-20T17:59:18.288 INFO:teuthology.orchestra.run.vm01.stderr:sudo: ntpd: command not found 2026-03-20T17:59:18.294 INFO:teuthology.orchestra.run.vm03.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-03-20T17:59:18.300 INFO:teuthology.orchestra.run.vm01.stdout:506 Cannot talk to daemon 2026-03-20T17:59:18.315 INFO:teuthology.orchestra.run.vm01.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-03-20T17:59:18.319 INFO:teuthology.orchestra.run.vm03.stderr:sudo: ntpd: command not found 2026-03-20T17:59:18.329 INFO:teuthology.orchestra.run.vm01.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-03-20T17:59:18.330 INFO:teuthology.orchestra.run.vm03.stdout:506 Cannot talk to daemon 2026-03-20T17:59:18.345 INFO:teuthology.orchestra.run.vm03.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-03-20T17:59:18.360 INFO:teuthology.orchestra.run.vm03.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-03-20T17:59:18.379 INFO:teuthology.orchestra.run.vm01.stderr:bash: line 1: ntpq: command not found 2026-03-20T17:59:18.381 INFO:teuthology.orchestra.run.vm01.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-03-20T17:59:18.381 INFO:teuthology.orchestra.run.vm01.stdout:=============================================================================== 2026-03-20T17:59:18.414 INFO:teuthology.orchestra.run.vm03.stderr:bash: line 1: ntpq: command not found 2026-03-20T17:59:18.416 INFO:teuthology.orchestra.run.vm03.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-03-20T17:59:18.416 INFO:teuthology.orchestra.run.vm03.stdout:=============================================================================== 2026-03-20T17:59:18.416 INFO:teuthology.run_tasks:Running task install... 2026-03-20T17:59:18.418 DEBUG:teuthology.task.install:project ceph 2026-03-20T17:59:18.418 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '70f8415b300f041766fa27faf7d5472699e32388'}, 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}} 2026-03-20T17:59:18.418 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '70f8415b300f041766fa27faf7d5472699e32388', 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}} 2026-03-20T17:59:18.418 INFO:teuthology.task.install:Using flavor: default 2026-03-20T17:59:18.420 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-03-20T17:59:18.420 INFO:teuthology.task.install:extra packages: [] 2026-03-20T17:59:18.420 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '70f8415b300f041766fa27faf7d5472699e32388', 'tag': None, 'wait_for_package': False} 2026-03-20T17:59:18.421 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=70f8415b300f041766fa27faf7d5472699e32388 2026-03-20T17:59:18.421 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '70f8415b300f041766fa27faf7d5472699e32388', 'tag': None, 'wait_for_package': False} 2026-03-20T17:59:18.421 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=70f8415b300f041766fa27faf7d5472699e32388 2026-03-20T17:59:19.040 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/tentacle/70f8415b300f041766fa27faf7d5472699e32388/centos/9/flavors/default/ 2026-03-20T17:59:19.040 INFO:teuthology.task.install.rpm:Package version is 20.2.0-712.g70f8415b 2026-03-20T17:59:19.096 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/tentacle/70f8415b300f041766fa27faf7d5472699e32388/centos/9/flavors/default/ 2026-03-20T17:59:19.096 INFO:teuthology.task.install.rpm:Package version is 20.2.0-712.g70f8415b 2026-03-20T17:59:19.666 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/tentacle/70f8415b300f041766fa27faf7d5472699e32388/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/tentacle/70f8415b300f041766fa27faf7d5472699e32388/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/tentacle/70f8415b300f041766fa27faf7d5472699e32388/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-03-20T17:59:19.666 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T17:59:19.666 DEBUG:teuthology.orchestra.run.vm03:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-03-20T17:59:19.692 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, bzip2, perl-Test-Harness, python3-jmespath, python3-xmltodict, s3cmd on remote rpm x86_64 2026-03-20T17:59:19.692 DEBUG:teuthology.orchestra.run.vm03:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/70f8415b300f041766fa27faf7d5472699e32388/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-03-20T17:59:19.720 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/tentacle/70f8415b300f041766fa27faf7d5472699e32388/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/tentacle/70f8415b300f041766fa27faf7d5472699e32388/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/tentacle/70f8415b300f041766fa27faf7d5472699e32388/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-03-20T17:59:19.720 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T17:59:19.720 DEBUG:teuthology.orchestra.run.vm01:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-03-20T17:59:19.748 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, bzip2, perl-Test-Harness, python3-jmespath, python3-xmltodict, s3cmd on remote rpm x86_64 2026-03-20T17:59:19.748 DEBUG:teuthology.orchestra.run.vm01:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/70f8415b300f041766fa27faf7d5472699e32388/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-03-20T17:59:19.763 DEBUG:teuthology.orchestra.run.vm03:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-03-20T17:59:19.817 DEBUG:teuthology.orchestra.run.vm01:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-03-20T17:59:19.845 DEBUG:teuthology.orchestra.run.vm03:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-03-20T17:59:19.895 DEBUG:teuthology.orchestra.run.vm01:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-03-20T17:59:19.912 INFO:teuthology.orchestra.run.vm03.stdout:check_obsoletes = 1 2026-03-20T17:59:19.913 DEBUG:teuthology.orchestra.run.vm03:> sudo yum clean all 2026-03-20T17:59:19.962 INFO:teuthology.orchestra.run.vm01.stdout:check_obsoletes = 1 2026-03-20T17:59:19.963 DEBUG:teuthology.orchestra.run.vm01:> sudo yum clean all 2026-03-20T17:59:20.086 INFO:teuthology.orchestra.run.vm03.stdout:41 files removed 2026-03-20T17:59:20.106 DEBUG:teuthology.orchestra.run.vm03:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd bzip2 perl-Test-Harness python3-jmespath python3-xmltodict s3cmd 2026-03-20T17:59:20.145 INFO:teuthology.orchestra.run.vm01.stdout:41 files removed 2026-03-20T17:59:20.167 DEBUG:teuthology.orchestra.run.vm01:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd bzip2 perl-Test-Harness python3-jmespath python3-xmltodict s3cmd 2026-03-20T17:59:21.549 INFO:teuthology.orchestra.run.vm03.stdout:ceph packages for x86_64 68 kB/s | 87 kB 00:01 2026-03-20T17:59:21.630 INFO:teuthology.orchestra.run.vm01.stdout:ceph packages for x86_64 67 kB/s | 87 kB 00:01 2026-03-20T17:59:22.959 INFO:teuthology.orchestra.run.vm03.stdout:ceph noarch packages 13 kB/s | 18 kB 00:01 2026-03-20T17:59:23.030 INFO:teuthology.orchestra.run.vm01.stdout:ceph noarch packages 13 kB/s | 18 kB 00:01 2026-03-20T17:59:24.278 INFO:teuthology.orchestra.run.vm03.stdout:ceph source packages 1.5 kB/s | 1.9 kB 00:01 2026-03-20T17:59:24.346 INFO:teuthology.orchestra.run.vm01.stdout:ceph source packages 1.5 kB/s | 1.9 kB 00:01 2026-03-20T17:59:25.826 INFO:teuthology.orchestra.run.vm03.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.9 MB 00:01 2026-03-20T17:59:27.704 INFO:teuthology.orchestra.run.vm03.stdout:CentOS Stream 9 - AppStream 21 MB/s | 27 MB 00:01 2026-03-20T17:59:28.503 INFO:teuthology.orchestra.run.vm01.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.9 MB 00:04 2026-03-20T17:59:30.792 INFO:teuthology.orchestra.run.vm01.stdout:CentOS Stream 9 - AppStream 16 MB/s | 27 MB 00:01 2026-03-20T17:59:31.712 INFO:teuthology.orchestra.run.vm03.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 8.0 MB 00:01 2026-03-20T17:59:33.310 INFO:teuthology.orchestra.run.vm03.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 20 kB 00:00 2026-03-20T17:59:33.872 INFO:teuthology.orchestra.run.vm03.stdout:Extra Packages for Enterprise Linux 42 MB/s | 20 MB 00:00 2026-03-20T17:59:35.731 INFO:teuthology.orchestra.run.vm01.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 8.0 MB 00:02 2026-03-20T17:59:37.555 INFO:teuthology.orchestra.run.vm01.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 20 kB 00:00 2026-03-20T17:59:38.509 INFO:teuthology.orchestra.run.vm01.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2026-03-20T17:59:38.537 INFO:teuthology.orchestra.run.vm03.stdout:lab-extras 60 kB/s | 50 kB 00:00 2026-03-20T17:59:39.983 INFO:teuthology.orchestra.run.vm03.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2026-03-20T17:59:39.984 INFO:teuthology.orchestra.run.vm03.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2026-03-20T17:59:40.018 INFO:teuthology.orchestra.run.vm03.stdout:Dependencies resolved. 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout:====================================================================================== 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: Package Arch Version Repository Size 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout:====================================================================================== 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout:Installing: 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: bzip2 x86_64 1.0.8-11.el9 baseos 55 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: ceph x86_64 2:20.2.0-712.g70f8415b.el9 ceph 6.5 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: ceph-base x86_64 2:20.2.0-712.g70f8415b.el9 ceph 5.9 M 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: ceph-fuse x86_64 2:20.2.0-712.g70f8415b.el9 ceph 939 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: ceph-immutable-object-cache x86_64 2:20.2.0-712.g70f8415b.el9 ceph 154 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mgr x86_64 2:20.2.0-712.g70f8415b.el9 ceph 962 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mgr-cephadm noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 173 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mgr-dashboard noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 11 M 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mgr-diskprediction-local noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 7.4 M 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mgr-rook noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 50 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: ceph-radosgw x86_64 2:20.2.0-712.g70f8415b.el9 ceph 24 M 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: ceph-test x86_64 2:20.2.0-712.g70f8415b.el9 ceph 84 M 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: ceph-volume noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 298 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: cephadm noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 1.0 M 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: libcephfs-devel x86_64 2:20.2.0-712.g70f8415b.el9 ceph 34 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: libcephfs2 x86_64 2:20.2.0-712.g70f8415b.el9 ceph 866 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: librados-devel x86_64 2:20.2.0-712.g70f8415b.el9 ceph 126 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: perl-Test-Harness noarch 1:3.42-461.el9 appstream 295 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: python3-cephfs x86_64 2:20.2.0-712.g70f8415b.el9 ceph 163 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: python3-rados x86_64 2:20.2.0-712.g70f8415b.el9 ceph 324 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: python3-rbd x86_64 2:20.2.0-712.g70f8415b.el9 ceph 304 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: python3-rgw x86_64 2:20.2.0-712.g70f8415b.el9 ceph 99 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: rbd-fuse x86_64 2:20.2.0-712.g70f8415b.el9 ceph 91 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: rbd-mirror x86_64 2:20.2.0-712.g70f8415b.el9 ceph 2.9 M 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: rbd-nbd x86_64 2:20.2.0-712.g70f8415b.el9 ceph 180 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: s3cmd noarch 2.4.0-1.el9 epel 206 k 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout:Upgrading: 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: librados2 x86_64 2:20.2.0-712.g70f8415b.el9 ceph 3.5 M 2026-03-20T17:59:40.023 INFO:teuthology.orchestra.run.vm03.stdout: librbd1 x86_64 2:20.2.0-712.g70f8415b.el9 ceph 2.8 M 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout:Installing dependencies: 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: ceph-common x86_64 2:20.2.0-712.g70f8415b.el9 ceph 24 M 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: ceph-grafana-dashboards noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 43 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mds x86_64 2:20.2.0-712.g70f8415b.el9 ceph 2.3 M 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mgr-modules-core noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 290 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mon x86_64 2:20.2.0-712.g70f8415b.el9 ceph 5.0 M 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: ceph-osd x86_64 2:20.2.0-712.g70f8415b.el9 ceph 17 M 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: ceph-prometheus-alerts noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 17 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: ceph-selinux x86_64 2:20.2.0-712.g70f8415b.el9 ceph 25 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: fuse x86_64 2.9.9-17.el9 baseos 80 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libcephfs-proxy2 x86_64 2:20.2.0-712.g70f8415b.el9 ceph 24 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libcephsqlite x86_64 2:20.2.0-712.g70f8415b.el9 ceph 164 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libradosstriper1 x86_64 2:20.2.0-712.g70f8415b.el9 ceph 250 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: librgw2 x86_64 2:20.2.0-712.g70f8415b.el9 ceph 6.4 M 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: perl-Benchmark noarch 1.23-483.el9 appstream 26 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: python3-ceph-argparse x86_64 2:20.2.0-712.g70f8415b.el9 ceph 45 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: python3-ceph-common x86_64 2:20.2.0-712.g70f8415b.el9 ceph 175 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-03-20T17:59:40.024 INFO:teuthology.orchestra.run.vm03.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout:Installing weak dependencies: 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout:Transaction Summary 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout:====================================================================================== 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout:Install 136 Packages 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout:Upgrade 2 Packages 2026-03-20T17:59:40.025 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T17:59:40.026 INFO:teuthology.orchestra.run.vm03.stdout:Total download size: 267 M 2026-03-20T17:59:40.026 INFO:teuthology.orchestra.run.vm03.stdout:Downloading Packages: 2026-03-20T17:59:40.983 INFO:teuthology.orchestra.run.vm03.stdout:(1/138): ceph-20.2.0-712.g70f8415b.el9.x86_64.r 14 kB/s | 6.5 kB 00:00 2026-03-20T17:59:41.787 INFO:teuthology.orchestra.run.vm03.stdout:(2/138): ceph-fuse-20.2.0-712.g70f8415b.el9.x86 1.1 MB/s | 939 kB 00:00 2026-03-20T17:59:41.905 INFO:teuthology.orchestra.run.vm03.stdout:(3/138): ceph-immutable-object-cache-20.2.0-712 1.3 MB/s | 154 kB 00:00 2026-03-20T17:59:42.711 INFO:teuthology.orchestra.run.vm03.stdout:(4/138): ceph-mds-20.2.0-712.g70f8415b.el9.x86_ 2.9 MB/s | 2.3 MB 00:00 2026-03-20T17:59:43.032 INFO:teuthology.orchestra.run.vm01.stdout:lab-extras 60 kB/s | 50 kB 00:00 2026-03-20T17:59:43.076 INFO:teuthology.orchestra.run.vm03.stdout:(5/138): ceph-mgr-20.2.0-712.g70f8415b.el9.x86_ 2.6 MB/s | 962 kB 00:00 2026-03-20T17:59:43.172 INFO:teuthology.orchestra.run.vm03.stdout:(6/138): ceph-base-20.2.0-712.g70f8415b.el9.x86 2.2 MB/s | 5.9 MB 00:02 2026-03-20T17:59:44.118 INFO:teuthology.orchestra.run.vm03.stdout:(7/138): ceph-mon-20.2.0-712.g70f8415b.el9.x86_ 4.8 MB/s | 5.0 MB 00:01 2026-03-20T17:59:44.353 INFO:teuthology.orchestra.run.vm01.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2026-03-20T17:59:44.353 INFO:teuthology.orchestra.run.vm01.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2026-03-20T17:59:44.386 INFO:teuthology.orchestra.run.vm01.stdout:Dependencies resolved. 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout:====================================================================================== 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: Package Arch Version Repository Size 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout:====================================================================================== 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout:Installing: 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: bzip2 x86_64 1.0.8-11.el9 baseos 55 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: ceph x86_64 2:20.2.0-712.g70f8415b.el9 ceph 6.5 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: ceph-base x86_64 2:20.2.0-712.g70f8415b.el9 ceph 5.9 M 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: ceph-fuse x86_64 2:20.2.0-712.g70f8415b.el9 ceph 939 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: ceph-immutable-object-cache x86_64 2:20.2.0-712.g70f8415b.el9 ceph 154 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mgr x86_64 2:20.2.0-712.g70f8415b.el9 ceph 962 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mgr-cephadm noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 173 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mgr-dashboard noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 11 M 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mgr-diskprediction-local noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 7.4 M 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mgr-rook noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 50 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: ceph-radosgw x86_64 2:20.2.0-712.g70f8415b.el9 ceph 24 M 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: ceph-test x86_64 2:20.2.0-712.g70f8415b.el9 ceph 84 M 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: ceph-volume noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 298 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: cephadm noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 1.0 M 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: libcephfs-devel x86_64 2:20.2.0-712.g70f8415b.el9 ceph 34 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: libcephfs2 x86_64 2:20.2.0-712.g70f8415b.el9 ceph 866 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: librados-devel x86_64 2:20.2.0-712.g70f8415b.el9 ceph 126 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: perl-Test-Harness noarch 1:3.42-461.el9 appstream 295 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: python3-cephfs x86_64 2:20.2.0-712.g70f8415b.el9 ceph 163 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: python3-rados x86_64 2:20.2.0-712.g70f8415b.el9 ceph 324 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: python3-rbd x86_64 2:20.2.0-712.g70f8415b.el9 ceph 304 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: python3-rgw x86_64 2:20.2.0-712.g70f8415b.el9 ceph 99 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: rbd-fuse x86_64 2:20.2.0-712.g70f8415b.el9 ceph 91 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: rbd-mirror x86_64 2:20.2.0-712.g70f8415b.el9 ceph 2.9 M 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: rbd-nbd x86_64 2:20.2.0-712.g70f8415b.el9 ceph 180 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: s3cmd noarch 2.4.0-1.el9 epel 206 k 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout:Upgrading: 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: librados2 x86_64 2:20.2.0-712.g70f8415b.el9 ceph 3.5 M 2026-03-20T17:59:44.390 INFO:teuthology.orchestra.run.vm01.stdout: librbd1 x86_64 2:20.2.0-712.g70f8415b.el9 ceph 2.8 M 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout:Installing dependencies: 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: ceph-common x86_64 2:20.2.0-712.g70f8415b.el9 ceph 24 M 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: ceph-grafana-dashboards noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 43 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mds x86_64 2:20.2.0-712.g70f8415b.el9 ceph 2.3 M 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mgr-modules-core noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 290 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mon x86_64 2:20.2.0-712.g70f8415b.el9 ceph 5.0 M 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: ceph-osd x86_64 2:20.2.0-712.g70f8415b.el9 ceph 17 M 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: ceph-prometheus-alerts noarch 2:20.2.0-712.g70f8415b.el9 ceph-noarch 17 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: ceph-selinux x86_64 2:20.2.0-712.g70f8415b.el9 ceph 25 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: fuse x86_64 2.9.9-17.el9 baseos 80 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libcephfs-proxy2 x86_64 2:20.2.0-712.g70f8415b.el9 ceph 24 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libcephsqlite x86_64 2:20.2.0-712.g70f8415b.el9 ceph 164 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libradosstriper1 x86_64 2:20.2.0-712.g70f8415b.el9 ceph 250 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: librgw2 x86_64 2:20.2.0-712.g70f8415b.el9 ceph 6.4 M 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: perl-Benchmark noarch 1.23-483.el9 appstream 26 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-ceph-argparse x86_64 2:20.2.0-712.g70f8415b.el9 ceph 45 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-ceph-common x86_64 2:20.2.0-712.g70f8415b.el9 ceph 175 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-03-20T17:59:44.391 INFO:teuthology.orchestra.run.vm01.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout:Installing weak dependencies: 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout:Transaction Summary 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout:====================================================================================== 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout:Install 136 Packages 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout:Upgrade 2 Packages 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout:Total download size: 267 M 2026-03-20T17:59:44.392 INFO:teuthology.orchestra.run.vm01.stdout:Downloading Packages: 2026-03-20T17:59:45.577 INFO:teuthology.orchestra.run.vm03.stdout:(8/138): ceph-osd-20.2.0-712.g70f8415b.el9.x86_ 7.1 MB/s | 17 MB 00:02 2026-03-20T17:59:45.712 INFO:teuthology.orchestra.run.vm03.stdout:(9/138): ceph-selinux-20.2.0-712.g70f8415b.el9. 186 kB/s | 25 kB 00:00 2026-03-20T17:59:46.060 INFO:teuthology.orchestra.run.vm01.stdout:(1/138): ceph-20.2.0-712.g70f8415b.el9.x86_64.r 13 kB/s | 6.5 kB 00:00 2026-03-20T17:59:46.462 INFO:teuthology.orchestra.run.vm03.stdout:(10/138): ceph-radosgw-20.2.0-712.g70f8415b.el9 10 MB/s | 24 MB 00:02 2026-03-20T17:59:46.578 INFO:teuthology.orchestra.run.vm03.stdout:(11/138): libcephfs-devel-20.2.0-712.g70f8415b. 296 kB/s | 34 kB 00:00 2026-03-20T17:59:46.694 INFO:teuthology.orchestra.run.vm03.stdout:(12/138): libcephfs-proxy2-20.2.0-712.g70f8415b 209 kB/s | 24 kB 00:00 2026-03-20T17:59:46.847 INFO:teuthology.orchestra.run.vm03.stdout:(13/138): libcephfs2-20.2.0-712.g70f8415b.el9.x 5.5 MB/s | 866 kB 00:00 2026-03-20T17:59:46.905 INFO:teuthology.orchestra.run.vm01.stdout:(2/138): ceph-fuse-20.2.0-712.g70f8415b.el9.x86 1.1 MB/s | 939 kB 00:00 2026-03-20T17:59:46.965 INFO:teuthology.orchestra.run.vm03.stdout:(14/138): libcephsqlite-20.2.0-712.g70f8415b.el 1.4 MB/s | 164 kB 00:00 2026-03-20T17:59:47.024 INFO:teuthology.orchestra.run.vm01.stdout:(3/138): ceph-immutable-object-cache-20.2.0-712 1.3 MB/s | 154 kB 00:00 2026-03-20T17:59:47.085 INFO:teuthology.orchestra.run.vm03.stdout:(15/138): librados-devel-20.2.0-712.g70f8415b.e 1.0 MB/s | 126 kB 00:00 2026-03-20T17:59:47.206 INFO:teuthology.orchestra.run.vm03.stdout:(16/138): libradosstriper1-20.2.0-712.g70f8415b 2.0 MB/s | 250 kB 00:00 2026-03-20T17:59:47.692 INFO:teuthology.orchestra.run.vm03.stdout:(17/138): librgw2-20.2.0-712.g70f8415b.el9.x86_ 13 MB/s | 6.4 MB 00:00 2026-03-20T17:59:47.811 INFO:teuthology.orchestra.run.vm03.stdout:(18/138): python3-ceph-argparse-20.2.0-712.g70f 380 kB/s | 45 kB 00:00 2026-03-20T17:59:47.851 INFO:teuthology.orchestra.run.vm01.stdout:(4/138): ceph-mds-20.2.0-712.g70f8415b.el9.x86_ 2.8 MB/s | 2.3 MB 00:00 2026-03-20T17:59:47.930 INFO:teuthology.orchestra.run.vm03.stdout:(19/138): python3-ceph-common-20.2.0-712.g70f84 1.4 MB/s | 175 kB 00:00 2026-03-20T17:59:48.081 INFO:teuthology.orchestra.run.vm03.stdout:(20/138): python3-cephfs-20.2.0-712.g70f8415b.e 1.1 MB/s | 163 kB 00:00 2026-03-20T17:59:48.155 INFO:teuthology.orchestra.run.vm03.stdout:(21/138): ceph-common-20.2.0-712.g70f8415b.el9. 3.1 MB/s | 24 MB 00:07 2026-03-20T17:59:48.227 INFO:teuthology.orchestra.run.vm03.stdout:(22/138): python3-rados-20.2.0-712.g70f8415b.el 2.2 MB/s | 324 kB 00:00 2026-03-20T17:59:48.231 INFO:teuthology.orchestra.run.vm01.stdout:(5/138): ceph-mgr-20.2.0-712.g70f8415b.el9.x86_ 2.5 MB/s | 962 kB 00:00 2026-03-20T17:59:48.310 INFO:teuthology.orchestra.run.vm03.stdout:(23/138): python3-rbd-20.2.0-712.g70f8415b.el9. 1.9 MB/s | 304 kB 00:00 2026-03-20T17:59:48.394 INFO:teuthology.orchestra.run.vm03.stdout:(24/138): python3-rgw-20.2.0-712.g70f8415b.el9. 593 kB/s | 99 kB 00:00 2026-03-20T17:59:48.439 INFO:teuthology.orchestra.run.vm03.stdout:(25/138): rbd-fuse-20.2.0-712.g70f8415b.el9.x86 706 kB/s | 91 kB 00:00 2026-03-20T17:59:48.569 INFO:teuthology.orchestra.run.vm03.stdout:(26/138): rbd-nbd-20.2.0-712.g70f8415b.el9.x86_ 1.4 MB/s | 180 kB 00:00 2026-03-20T17:59:48.681 INFO:teuthology.orchestra.run.vm03.stdout:(27/138): rbd-mirror-20.2.0-712.g70f8415b.el9.x 10 MB/s | 2.9 MB 00:00 2026-03-20T17:59:48.696 INFO:teuthology.orchestra.run.vm03.stdout:(28/138): ceph-grafana-dashboards-20.2.0-712.g7 339 kB/s | 43 kB 00:00 2026-03-20T17:59:48.812 INFO:teuthology.orchestra.run.vm03.stdout:(29/138): ceph-mgr-cephadm-20.2.0-712.g70f8415b 1.3 MB/s | 173 kB 00:00 2026-03-20T17:59:48.813 INFO:teuthology.orchestra.run.vm01.stdout:(6/138): ceph-base-20.2.0-712.g70f8415b.el9.x86 1.8 MB/s | 5.9 MB 00:03 2026-03-20T17:59:49.367 INFO:teuthology.orchestra.run.vm03.stdout:(30/138): ceph-mgr-diskprediction-local-20.2.0- 13 MB/s | 7.4 MB 00:00 2026-03-20T17:59:49.487 INFO:teuthology.orchestra.run.vm03.stdout:(31/138): ceph-mgr-modules-core-20.2.0-712.g70f 2.4 MB/s | 290 kB 00:00 2026-03-20T17:59:49.605 INFO:teuthology.orchestra.run.vm03.stdout:(32/138): ceph-mgr-rook-20.2.0-712.g70f8415b.el 426 kB/s | 50 kB 00:00 2026-03-20T17:59:49.720 INFO:teuthology.orchestra.run.vm03.stdout:(33/138): ceph-prometheus-alerts-20.2.0-712.g70 151 kB/s | 17 kB 00:00 2026-03-20T17:59:49.825 INFO:teuthology.orchestra.run.vm01.stdout:(7/138): ceph-mon-20.2.0-712.g70f8415b.el9.x86_ 3.2 MB/s | 5.0 MB 00:01 2026-03-20T17:59:49.840 INFO:teuthology.orchestra.run.vm03.stdout:(34/138): ceph-volume-20.2.0-712.g70f8415b.el9. 2.4 MB/s | 298 kB 00:00 2026-03-20T17:59:49.970 INFO:teuthology.orchestra.run.vm03.stdout:(35/138): cephadm-20.2.0-712.g70f8415b.el9.noar 7.7 MB/s | 1.0 MB 00:00 2026-03-20T17:59:50.124 INFO:teuthology.orchestra.run.vm03.stdout:(36/138): bzip2-1.0.8-11.el9.x86_64.rpm 357 kB/s | 55 kB 00:00 2026-03-20T17:59:50.176 INFO:teuthology.orchestra.run.vm01.stdout:(8/138): ceph-common-20.2.0-712.g70f8415b.el9.x 5.1 MB/s | 24 MB 00:04 2026-03-20T17:59:50.208 INFO:teuthology.orchestra.run.vm03.stdout:(37/138): cryptsetup-2.8.1-3.el9.x86_64.rpm 4.1 MB/s | 351 kB 00:00 2026-03-20T17:59:50.234 INFO:teuthology.orchestra.run.vm03.stdout:(38/138): fuse-2.9.9-17.el9.x86_64.rpm 3.1 MB/s | 80 kB 00:00 2026-03-20T17:59:50.259 INFO:teuthology.orchestra.run.vm03.stdout:(39/138): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.6 MB/s | 40 kB 00:00 2026-03-20T17:59:50.285 INFO:teuthology.orchestra.run.vm03.stdout:(40/138): libconfig-1.7.2-9.el9.x86_64.rpm 2.7 MB/s | 72 kB 00:00 2026-03-20T17:59:50.287 INFO:teuthology.orchestra.run.vm01.stdout:(9/138): ceph-selinux-20.2.0-712.g70f8415b.el9. 226 kB/s | 25 kB 00:00 2026-03-20T17:59:50.385 INFO:teuthology.orchestra.run.vm03.stdout:(41/138): libgfortran-11.5.0-14.el9.x86_64.rpm 7.8 MB/s | 794 kB 00:00 2026-03-20T17:59:50.436 INFO:teuthology.orchestra.run.vm03.stdout:(42/138): libquadmath-11.5.0-14.el9.x86_64.rpm 3.6 MB/s | 184 kB 00:00 2026-03-20T17:59:50.468 INFO:teuthology.orchestra.run.vm03.stdout:(43/138): mailcap-2.1.49-5.el9.noarch.rpm 1.0 MB/s | 33 kB 00:00 2026-03-20T17:59:50.495 INFO:teuthology.orchestra.run.vm03.stdout:(44/138): pciutils-3.7.0-7.el9.x86_64.rpm 3.4 MB/s | 93 kB 00:00 2026-03-20T17:59:50.524 INFO:teuthology.orchestra.run.vm03.stdout:(45/138): python3-cffi-1.14.5-5.el9.x86_64.rpm 8.6 MB/s | 253 kB 00:00 2026-03-20T17:59:50.569 INFO:teuthology.orchestra.run.vm03.stdout:(46/138): python3-cryptography-36.0.1-5.el9.x86 28 MB/s | 1.2 MB 00:00 2026-03-20T17:59:50.595 INFO:teuthology.orchestra.run.vm03.stdout:(47/138): python3-ply-3.11-14.el9.noarch.rpm 4.0 MB/s | 106 kB 00:00 2026-03-20T17:59:50.623 INFO:teuthology.orchestra.run.vm03.stdout:(48/138): python3-pycparser-2.20-6.el9.noarch.r 4.7 MB/s | 135 kB 00:00 2026-03-20T17:59:50.648 INFO:teuthology.orchestra.run.vm03.stdout:(49/138): python3-pyparsing-2.4.7-9.el9.noarch. 6.0 MB/s | 150 kB 00:00 2026-03-20T17:59:50.680 INFO:teuthology.orchestra.run.vm03.stdout:(50/138): python3-requests-2.25.1-10.el9.noarch 4.0 MB/s | 126 kB 00:00 2026-03-20T17:59:50.705 INFO:teuthology.orchestra.run.vm03.stdout:(51/138): python3-urllib3-1.26.5-7.el9.noarch.r 8.5 MB/s | 218 kB 00:00 2026-03-20T17:59:50.730 INFO:teuthology.orchestra.run.vm03.stdout:(52/138): unzip-6.0-59.el9.x86_64.rpm 7.2 MB/s | 182 kB 00:00 2026-03-20T17:59:50.759 INFO:teuthology.orchestra.run.vm03.stdout:(53/138): zip-3.0-35.el9.x86_64.rpm 9.2 MB/s | 266 kB 00:00 2026-03-20T17:59:51.355 INFO:teuthology.orchestra.run.vm03.stdout:(54/138): ceph-mgr-dashboard-20.2.0-712.g70f841 4.0 MB/s | 11 MB 00:02 2026-03-20T17:59:51.943 INFO:teuthology.orchestra.run.vm03.stdout:(55/138): flexiblas-3.0.4-9.el9.x86_64.rpm 50 kB/s | 30 kB 00:00 2026-03-20T17:59:52.012 INFO:teuthology.orchestra.run.vm03.stdout:(56/138): boost-program-options-1.75.0-13.el9.x 83 kB/s | 104 kB 00:01 2026-03-20T17:59:52.051 INFO:teuthology.orchestra.run.vm03.stdout:(57/138): flexiblas-openblas-openmp-3.0.4-9.el9 384 kB/s | 15 kB 00:00 2026-03-20T17:59:52.127 INFO:teuthology.orchestra.run.vm03.stdout:(58/138): libnbd-1.20.3-4.el9.x86_64.rpm 2.1 MB/s | 164 kB 00:00 2026-03-20T17:59:52.167 INFO:teuthology.orchestra.run.vm03.stdout:(59/138): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.9 MB/s | 160 kB 00:00 2026-03-20T17:59:52.206 INFO:teuthology.orchestra.run.vm03.stdout:(60/138): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2026-03-20T17:59:52.271 INFO:teuthology.orchestra.run.vm03.stdout:(61/138): flexiblas-netlib-3.0.4-9.el9.x86_64.r 9.1 MB/s | 3.0 MB 00:00 2026-03-20T17:59:52.287 INFO:teuthology.orchestra.run.vm03.stdout:(62/138): librdkafka-1.6.1-102.el9.x86_64.rpm 8.0 MB/s | 662 kB 00:00 2026-03-20T17:59:52.315 INFO:teuthology.orchestra.run.vm03.stdout:(63/138): libstoragemgmt-1.10.1-1.el9.x86_64.rp 5.6 MB/s | 246 kB 00:00 2026-03-20T17:59:52.328 INFO:teuthology.orchestra.run.vm03.stdout:(64/138): libxslt-1.1.34-12.el9.x86_64.rpm 5.6 MB/s | 233 kB 00:00 2026-03-20T17:59:52.358 INFO:teuthology.orchestra.run.vm03.stdout:(65/138): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 292 kB 00:00 2026-03-20T17:59:52.369 INFO:teuthology.orchestra.run.vm03.stdout:(66/138): lua-5.4.4-4.el9.x86_64.rpm 4.6 MB/s | 188 kB 00:00 2026-03-20T17:59:52.398 INFO:teuthology.orchestra.run.vm03.stdout:(67/138): openblas-0.3.29-1.el9.x86_64.rpm 1.0 MB/s | 42 kB 00:00 2026-03-20T17:59:52.440 INFO:teuthology.orchestra.run.vm03.stdout:(68/138): perl-Benchmark-1.23-483.el9.noarch.rp 640 kB/s | 26 kB 00:00 2026-03-20T17:59:52.483 INFO:teuthology.orchestra.run.vm03.stdout:(69/138): perl-Test-Harness-3.42-461.el9.noarch 6.7 MB/s | 295 kB 00:00 2026-03-20T17:59:52.607 INFO:teuthology.orchestra.run.vm03.stdout:(70/138): protobuf-3.14.0-17.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2026-03-20T17:59:52.758 INFO:teuthology.orchestra.run.vm03.stdout:(71/138): openblas-openmp-0.3.29-1.el9.x86_64.r 14 MB/s | 5.3 MB 00:00 2026-03-20T17:59:52.803 INFO:teuthology.orchestra.run.vm03.stdout:(72/138): python3-devel-3.9.25-3.el9.x86_64.rpm 5.3 MB/s | 244 kB 00:00 2026-03-20T17:59:52.845 INFO:teuthology.orchestra.run.vm03.stdout:(73/138): python3-jinja2-2.11.3-8.el9.noarch.rp 5.9 MB/s | 249 kB 00:00 2026-03-20T17:59:52.884 INFO:teuthology.orchestra.run.vm03.stdout:(74/138): python3-jmespath-1.0.1-1.el9.noarch.r 1.2 MB/s | 48 kB 00:00 2026-03-20T17:59:52.924 INFO:teuthology.orchestra.run.vm03.stdout:(75/138): python3-libstoragemgmt-1.10.1-1.el9.x 4.3 MB/s | 177 kB 00:00 2026-03-20T17:59:52.963 INFO:teuthology.orchestra.run.vm03.stdout:(76/138): python3-markupsafe-1.1.1-12.el9.x86_6 905 kB/s | 35 kB 00:00 2026-03-20T17:59:53.215 INFO:teuthology.orchestra.run.vm03.stdout:(77/138): python3-babel-2.9.1-2.el9.noarch.rpm 9.8 MB/s | 6.0 MB 00:00 2026-03-20T17:59:53.295 INFO:teuthology.orchestra.run.vm03.stdout:(78/138): python3-numpy-f2py-1.23.5-2.el9.x86_6 5.4 MB/s | 442 kB 00:00 2026-03-20T17:59:53.336 INFO:teuthology.orchestra.run.vm03.stdout:(79/138): python3-packaging-20.9-5.el9.noarch.r 1.9 MB/s | 77 kB 00:00 2026-03-20T17:59:53.380 INFO:teuthology.orchestra.run.vm03.stdout:(80/138): python3-protobuf-3.14.0-17.el9.noarch 6.0 MB/s | 267 kB 00:00 2026-03-20T17:59:53.423 INFO:teuthology.orchestra.run.vm03.stdout:(81/138): python3-pyasn1-0.4.8-7.el9.noarch.rpm 3.6 MB/s | 157 kB 00:00 2026-03-20T17:59:53.624 INFO:teuthology.orchestra.run.vm03.stdout:(82/138): ceph-test-20.2.0-712.g70f8415b.el9.x8 11 MB/s | 84 MB 00:07 2026-03-20T17:59:53.626 INFO:teuthology.orchestra.run.vm03.stdout:(83/138): python3-pyasn1-modules-0.4.8-7.el9.no 1.3 MB/s | 277 kB 00:00 2026-03-20T17:59:53.642 INFO:teuthology.orchestra.run.vm03.stdout:(84/138): python3-numpy-1.23.5-2.el9.x86_64.rpm 9.0 MB/s | 6.1 MB 00:00 2026-03-20T17:59:53.681 INFO:teuthology.orchestra.run.vm03.stdout:(85/138): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2026-03-20T17:59:53.723 INFO:teuthology.orchestra.run.vm03.stdout:(86/138): qatlib-25.08.0-2.el9.x86_64.rpm 5.7 MB/s | 240 kB 00:00 2026-03-20T17:59:53.761 INFO:teuthology.orchestra.run.vm03.stdout:(87/138): qatlib-service-25.08.0-2.el9.x86_64.r 960 kB/s | 37 kB 00:00 2026-03-20T17:59:53.773 INFO:teuthology.orchestra.run.vm03.stdout:(88/138): python3-requests-oauthlib-1.3.0-12.el 360 kB/s | 54 kB 00:00 2026-03-20T17:59:53.801 INFO:teuthology.orchestra.run.vm03.stdout:(89/138): qatzip-libs-1.3.1-1.el9.x86_64.rpm 1.7 MB/s | 66 kB 00:00 2026-03-20T17:59:53.840 INFO:teuthology.orchestra.run.vm03.stdout:(90/138): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.6 MB/s | 64 kB 00:00 2026-03-20T17:59:53.887 INFO:teuthology.orchestra.run.vm03.stdout:(91/138): socat-1.7.4.1-8.el9.x86_64.rpm 2.6 MB/s | 303 kB 00:00 2026-03-20T17:59:53.986 INFO:teuthology.orchestra.run.vm03.stdout:(92/138): lua-devel-5.4.4-4.el9.x86_64.rpm 152 kB/s | 22 kB 00:00 2026-03-20T17:59:54.002 INFO:teuthology.orchestra.run.vm03.stdout:(93/138): abseil-cpp-20211102.0-4.el9.x86_64.rp 35 MB/s | 551 kB 00:00 2026-03-20T17:59:54.010 INFO:teuthology.orchestra.run.vm03.stdout:(94/138): gperftools-libs-2.9.1-3.el9.x86_64.rp 39 MB/s | 308 kB 00:00 2026-03-20T17:59:54.013 INFO:teuthology.orchestra.run.vm03.stdout:(95/138): grpc-data-1.46.7-10.el9.noarch.rpm 7.9 MB/s | 19 kB 00:00 2026-03-20T17:59:54.081 INFO:teuthology.orchestra.run.vm03.stdout:(96/138): libarrow-9.0.0-15.el9.x86_64.rpm 66 MB/s | 4.4 MB 00:00 2026-03-20T17:59:54.084 INFO:teuthology.orchestra.run.vm03.stdout:(97/138): libarrow-doc-9.0.0-15.el9.noarch.rpm 8.4 MB/s | 25 kB 00:00 2026-03-20T17:59:54.087 INFO:teuthology.orchestra.run.vm03.stdout:(98/138): liboath-2.6.12-1.el9.x86_64.rpm 14 MB/s | 49 kB 00:00 2026-03-20T17:59:54.091 INFO:teuthology.orchestra.run.vm03.stdout:(99/138): libunwind-1.6.2-1.el9.x86_64.rpm 19 MB/s | 67 kB 00:00 2026-03-20T17:59:54.096 INFO:teuthology.orchestra.run.vm03.stdout:(100/138): luarocks-3.9.2-5.el9.noarch.rpm 31 MB/s | 151 kB 00:00 2026-03-20T17:59:54.109 INFO:teuthology.orchestra.run.vm03.stdout:(101/138): parquet-libs-9.0.0-15.el9.x86_64.rpm 63 MB/s | 838 kB 00:00 2026-03-20T17:59:54.196 INFO:teuthology.orchestra.run.vm03.stdout:(102/138): python3-asyncssh-2.13.2-5.el9.noarch 6.2 MB/s | 548 kB 00:00 2026-03-20T17:59:54.203 INFO:teuthology.orchestra.run.vm03.stdout:(103/138): python3-autocommand-2.2.2-8.el9.noar 4.4 MB/s | 29 kB 00:00 2026-03-20T17:59:54.208 INFO:teuthology.orchestra.run.vm03.stdout:(104/138): python3-backports-tarfile-1.2.0-1.el 12 MB/s | 60 kB 00:00 2026-03-20T17:59:54.212 INFO:teuthology.orchestra.run.vm03.stdout:(105/138): python3-bcrypt-3.2.2-1.el9.x86_64.rp 12 MB/s | 43 kB 00:00 2026-03-20T17:59:54.215 INFO:teuthology.orchestra.run.vm03.stdout:(106/138): python3-cachetools-4.2.4-1.el9.noarc 10 MB/s | 32 kB 00:00 2026-03-20T17:59:54.217 INFO:teuthology.orchestra.run.vm03.stdout:(107/138): python3-certifi-2023.05.07-4.el9.noa 6.6 MB/s | 14 kB 00:00 2026-03-20T17:59:54.222 INFO:teuthology.orchestra.run.vm03.stdout:(108/138): python3-cheroot-10.0.1-4.el9.noarch. 39 MB/s | 173 kB 00:00 2026-03-20T17:59:54.229 INFO:teuthology.orchestra.run.vm03.stdout:(109/138): python3-cherrypy-18.6.1-2.el9.noarch 52 MB/s | 358 kB 00:00 2026-03-20T17:59:54.235 INFO:teuthology.orchestra.run.vm03.stdout:(110/138): python3-google-auth-2.45.0-1.el9.noa 42 MB/s | 254 kB 00:00 2026-03-20T17:59:54.268 INFO:teuthology.orchestra.run.vm03.stdout:(111/138): python3-grpcio-1.46.7-10.el9.x86_64. 62 MB/s | 2.0 MB 00:00 2026-03-20T17:59:54.273 INFO:teuthology.orchestra.run.vm03.stdout:(112/138): python3-grpcio-tools-1.46.7-10.el9.x 34 MB/s | 144 kB 00:00 2026-03-20T17:59:54.276 INFO:teuthology.orchestra.run.vm03.stdout:(113/138): python3-jaraco-8.2.1-3.el9.noarch.rp 3.9 MB/s | 11 kB 00:00 2026-03-20T17:59:54.279 INFO:teuthology.orchestra.run.vm03.stdout:(114/138): protobuf-compiler-3.14.0-17.el9.x86_ 2.1 MB/s | 862 kB 00:00 2026-03-20T17:59:54.280 INFO:teuthology.orchestra.run.vm03.stdout:(115/138): python3-jaraco-classes-3.2.1-5.el9.n 4.1 MB/s | 18 kB 00:00 2026-03-20T17:59:54.282 INFO:teuthology.orchestra.run.vm03.stdout:(116/138): python3-jaraco-context-6.0.1-3.el9.n 8.4 MB/s | 20 kB 00:00 2026-03-20T17:59:54.284 INFO:teuthology.orchestra.run.vm03.stdout:(117/138): python3-jaraco-collections-3.0.0-8.e 5.5 MB/s | 23 kB 00:00 2026-03-20T17:59:54.284 INFO:teuthology.orchestra.run.vm03.stdout:(118/138): python3-jaraco-functools-3.5.0-2.el9 9.4 MB/s | 19 kB 00:00 2026-03-20T17:59:54.287 INFO:teuthology.orchestra.run.vm03.stdout:(119/138): python3-jaraco-text-4.0.0-2.el9.noar 8.5 MB/s | 26 kB 00:00 2026-03-20T17:59:54.301 INFO:teuthology.orchestra.run.vm03.stdout:(120/138): python3-more-itertools-8.12.0-2.el9. 5.6 MB/s | 79 kB 00:00 2026-03-20T17:59:54.307 INFO:teuthology.orchestra.run.vm03.stdout:(121/138): python3-kubernetes-26.1.0-3.el9.noar 47 MB/s | 1.0 MB 00:00 2026-03-20T17:59:54.307 INFO:teuthology.orchestra.run.vm03.stdout:(122/138): python3-natsort-7.1.1-5.el9.noarch.r 9.3 MB/s | 58 kB 00:00 2026-03-20T17:59:54.309 INFO:teuthology.orchestra.run.vm03.stdout:(123/138): python3-portend-3.1.0-2.el9.noarch.r 6.8 MB/s | 16 kB 00:00 2026-03-20T17:59:54.312 INFO:teuthology.orchestra.run.vm03.stdout:(124/138): python3-pyOpenSSL-21.0.0-1.el9.noarc 20 MB/s | 90 kB 00:00 2026-03-20T17:59:54.313 INFO:teuthology.orchestra.run.vm03.stdout:(125/138): python3-repoze-lru-0.7-16.el9.noarch 7.6 MB/s | 31 kB 00:00 2026-03-20T17:59:54.317 INFO:teuthology.orchestra.run.vm03.stdout:(126/138): python3-rsa-4.9-2.el9.noarch.rpm 16 MB/s | 59 kB 00:00 2026-03-20T17:59:54.320 INFO:teuthology.orchestra.run.vm03.stdout:(127/138): python3-routes-2.5.1-5.el9.noarch.rp 23 MB/s | 188 kB 00:00 2026-03-20T17:59:54.321 INFO:teuthology.orchestra.run.vm03.stdout:(128/138): python3-tempora-5.0.0-2.el9.noarch.r 9.8 MB/s | 36 kB 00:00 2026-03-20T17:59:54.324 INFO:teuthology.orchestra.run.vm03.stdout:(129/138): python3-websocket-client-1.2.3-2.el9 24 MB/s | 90 kB 00:00 2026-03-20T17:59:54.326 INFO:teuthology.orchestra.run.vm03.stdout:(130/138): python3-typing-extensions-4.15.0-1.e 15 MB/s | 86 kB 00:00 2026-03-20T17:59:54.327 INFO:teuthology.orchestra.run.vm03.stdout:(131/138): python3-xmltodict-0.12.0-15.el9.noar 10 MB/s | 22 kB 00:00 2026-03-20T17:59:54.329 INFO:teuthology.orchestra.run.vm03.stdout:(132/138): python3-zc-lockfile-2.0-10.el9.noarc 6.4 MB/s | 20 kB 00:00 2026-03-20T17:59:54.336 INFO:teuthology.orchestra.run.vm03.stdout:(133/138): s3cmd-2.4.0-1.el9.noarch.rpm 28 MB/s | 206 kB 00:00 2026-03-20T17:59:54.338 INFO:teuthology.orchestra.run.vm03.stdout:(134/138): re2-20211101-20.el9.x86_64.rpm 17 MB/s | 191 kB 00:00 2026-03-20T17:59:54.364 INFO:teuthology.orchestra.run.vm03.stdout:(135/138): thrift-0.15.0-4.el9.x86_64.rpm 58 MB/s | 1.6 MB 00:00 2026-03-20T17:59:55.359 INFO:teuthology.orchestra.run.vm03.stdout:(136/138): librbd1-20.2.0-712.g70f8415b.el9.x86 2.9 MB/s | 2.8 MB 00:00 2026-03-20T17:59:55.946 INFO:teuthology.orchestra.run.vm01.stdout:(10/138): ceph-radosgw-20.2.0-712.g70f8415b.el9 3.9 MB/s | 24 MB 00:06 2026-03-20T17:59:55.989 INFO:teuthology.orchestra.run.vm03.stdout:(137/138): python3-scipy-1.9.3-2.el9.x86_64.rpm 8.2 MB/s | 19 MB 00:02 2026-03-20T17:59:56.001 INFO:teuthology.orchestra.run.vm03.stdout:(138/138): librados2-20.2.0-712.g70f8415b.el9.x 2.1 MB/s | 3.5 MB 00:01 2026-03-20T17:59:56.003 INFO:teuthology.orchestra.run.vm03.stdout:-------------------------------------------------------------------------------- 2026-03-20T17:59:56.003 INFO:teuthology.orchestra.run.vm03.stdout:Total 17 MB/s | 267 MB 00:15 2026-03-20T17:59:56.091 INFO:teuthology.orchestra.run.vm01.stdout:(11/138): libcephfs-devel-20.2.0-712.g70f8415b. 238 kB/s | 34 kB 00:00 2026-03-20T17:59:56.209 INFO:teuthology.orchestra.run.vm01.stdout:(12/138): libcephfs-proxy2-20.2.0-712.g70f8415b 205 kB/s | 24 kB 00:00 2026-03-20T17:59:56.451 INFO:teuthology.orchestra.run.vm01.stdout:(13/138): libcephfs2-20.2.0-712.g70f8415b.el9.x 3.5 MB/s | 866 kB 00:00 2026-03-20T17:59:56.571 INFO:teuthology.orchestra.run.vm01.stdout:(14/138): libcephsqlite-20.2.0-712.g70f8415b.el 1.3 MB/s | 164 kB 00:00 2026-03-20T17:59:56.631 INFO:teuthology.orchestra.run.vm03.stdout:Running transaction check 2026-03-20T17:59:56.691 INFO:teuthology.orchestra.run.vm01.stdout:(15/138): librados-devel-20.2.0-712.g70f8415b.e 1.0 MB/s | 126 kB 00:00 2026-03-20T17:59:56.694 INFO:teuthology.orchestra.run.vm03.stdout:Transaction check succeeded. 2026-03-20T17:59:56.695 INFO:teuthology.orchestra.run.vm03.stdout:Running transaction test 2026-03-20T17:59:56.775 INFO:teuthology.orchestra.run.vm01.stdout:(16/138): ceph-osd-20.2.0-712.g70f8415b.el9.x86 2.1 MB/s | 17 MB 00:07 2026-03-20T17:59:56.813 INFO:teuthology.orchestra.run.vm01.stdout:(17/138): libradosstriper1-20.2.0-712.g70f8415b 2.0 MB/s | 250 kB 00:00 2026-03-20T17:59:56.931 INFO:teuthology.orchestra.run.vm01.stdout:(18/138): python3-ceph-argparse-20.2.0-712.g70f 381 kB/s | 45 kB 00:00 2026-03-20T17:59:57.051 INFO:teuthology.orchestra.run.vm01.stdout:(19/138): python3-ceph-common-20.2.0-712.g70f84 1.4 MB/s | 175 kB 00:00 2026-03-20T17:59:57.172 INFO:teuthology.orchestra.run.vm01.stdout:(20/138): python3-cephfs-20.2.0-712.g70f8415b.e 1.3 MB/s | 163 kB 00:00 2026-03-20T17:59:57.296 INFO:teuthology.orchestra.run.vm01.stdout:(21/138): python3-rados-20.2.0-712.g70f8415b.el 2.6 MB/s | 324 kB 00:00 2026-03-20T17:59:57.418 INFO:teuthology.orchestra.run.vm01.stdout:(22/138): python3-rbd-20.2.0-712.g70f8415b.el9. 2.4 MB/s | 304 kB 00:00 2026-03-20T17:59:57.538 INFO:teuthology.orchestra.run.vm01.stdout:(23/138): python3-rgw-20.2.0-712.g70f8415b.el9. 829 kB/s | 99 kB 00:00 2026-03-20T17:59:57.659 INFO:teuthology.orchestra.run.vm01.stdout:(24/138): rbd-fuse-20.2.0-712.g70f8415b.el9.x86 756 kB/s | 91 kB 00:00 2026-03-20T17:59:57.735 INFO:teuthology.orchestra.run.vm03.stdout:Transaction test succeeded. 2026-03-20T17:59:57.735 INFO:teuthology.orchestra.run.vm03.stdout:Running transaction 2026-03-20T17:59:58.373 INFO:teuthology.orchestra.run.vm01.stdout:(25/138): rbd-mirror-20.2.0-712.g70f8415b.el9.x 4.1 MB/s | 2.9 MB 00:00 2026-03-20T17:59:58.494 INFO:teuthology.orchestra.run.vm01.stdout:(26/138): rbd-nbd-20.2.0-712.g70f8415b.el9.x86_ 1.5 MB/s | 180 kB 00:00 2026-03-20T17:59:58.613 INFO:teuthology.orchestra.run.vm01.stdout:(27/138): ceph-grafana-dashboards-20.2.0-712.g7 363 kB/s | 43 kB 00:00 2026-03-20T17:59:58.733 INFO:teuthology.orchestra.run.vm01.stdout:(28/138): ceph-mgr-cephadm-20.2.0-712.g70f8415b 1.4 MB/s | 173 kB 00:00 2026-03-20T17:59:58.906 INFO:teuthology.orchestra.run.vm03.stdout: Preparing : 1/1 2026-03-20T17:59:58.915 INFO:teuthology.orchestra.run.vm03.stdout: Installing : thrift-0.15.0-4.el9.x86_64 1/140 2026-03-20T17:59:58.918 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/140 2026-03-20T17:59:58.930 INFO:teuthology.orchestra.run.vm03.stdout: Installing : liboath-2.6.12-1.el9.x86_64 3/140 2026-03-20T17:59:59.112 INFO:teuthology.orchestra.run.vm03.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/140 2026-03-20T17:59:59.114 INFO:teuthology.orchestra.run.vm03.stdout: Upgrading : librados2-2:20.2.0-712.g70f8415b.el9.x86_64 5/140 2026-03-20T17:59:59.146 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: librados2-2:20.2.0-712.g70f8415b.el9.x86_64 5/140 2026-03-20T17:59:59.156 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-rados-2:20.2.0-712.g70f8415b.el9.x86_64 6/140 2026-03-20T17:59:59.160 INFO:teuthology.orchestra.run.vm03.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 7/140 2026-03-20T17:59:59.164 INFO:teuthology.orchestra.run.vm03.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 8/140 2026-03-20T17:59:59.167 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 9/140 2026-03-20T17:59:59.173 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 10/140 2026-03-20T17:59:59.286 INFO:teuthology.orchestra.run.vm01.stdout:(29/138): ceph-test-20.2.0-712.g70f8415b.el9.x8 9.3 MB/s | 84 MB 00:08 2026-03-20T17:59:59.305 INFO:teuthology.orchestra.run.vm01.stdout:(30/138): librgw2-20.2.0-712.g70f8415b.el9.x86_ 2.5 MB/s | 6.4 MB 00:02 2026-03-20T17:59:59.313 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 11/140 2026-03-20T17:59:59.315 INFO:teuthology.orchestra.run.vm03.stdout: Upgrading : librbd1-2:20.2.0-712.g70f8415b.el9.x86_64 12/140 2026-03-20T17:59:59.338 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: librbd1-2:20.2.0-712.g70f8415b.el9.x86_64 12/140 2026-03-20T17:59:59.339 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libcephsqlite-2:20.2.0-712.g70f8415b.el9.x86_64 13/140 2026-03-20T17:59:59.363 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: libcephsqlite-2:20.2.0-712.g70f8415b.el9.x86_64 13/140 2026-03-20T17:59:59.366 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libradosstriper1-2:20.2.0-712.g70f8415b.el9.x86_ 14/140 2026-03-20T17:59:59.382 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: libradosstriper1-2:20.2.0-712.g70f8415b.el9.x86_ 14/140 2026-03-20T17:59:59.416 INFO:teuthology.orchestra.run.vm03.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/140 2026-03-20T17:59:59.420 INFO:teuthology.orchestra.run.vm01.stdout:(31/138): ceph-mgr-modules-core-20.2.0-712.g70f 2.5 MB/s | 290 kB 00:00 2026-03-20T17:59:59.442 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/140 2026-03-20T17:59:59.453 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 17/140 2026-03-20T17:59:59.460 INFO:teuthology.orchestra.run.vm03.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 18/140 2026-03-20T17:59:59.464 INFO:teuthology.orchestra.run.vm03.stdout: Installing : lua-5.4.4-4.el9.x86_64 19/140 2026-03-20T17:59:59.470 INFO:teuthology.orchestra.run.vm03.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 20/140 2026-03-20T17:59:59.500 INFO:teuthology.orchestra.run.vm03.stdout: Installing : unzip-6.0-59.el9.x86_64 21/140 2026-03-20T17:59:59.519 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 22/140 2026-03-20T17:59:59.524 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 23/140 2026-03-20T17:59:59.532 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 24/140 2026-03-20T17:59:59.534 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 25/140 2026-03-20T17:59:59.556 INFO:teuthology.orchestra.run.vm01.stdout:(32/138): ceph-mgr-rook-20.2.0-712.g70f8415b.el 370 kB/s | 50 kB 00:00 2026-03-20T17:59:59.574 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 26/140 2026-03-20T17:59:59.581 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-ceph-common-2:20.2.0-712.g70f8415b.el9.x 27/140 2026-03-20T17:59:59.584 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-ceph-argparse-2:20.2.0-712.g70f8415b.el9 28/140 2026-03-20T17:59:59.585 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libcephfs-proxy2-2:20.2.0-712.g70f8415b.el9.x86_ 29/140 2026-03-20T17:59:59.638 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: libcephfs-proxy2-2:20.2.0-712.g70f8415b.el9.x86_ 29/140 2026-03-20T17:59:59.640 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libcephfs2-2:20.2.0-712.g70f8415b.el9.x86_64 30/140 2026-03-20T17:59:59.662 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: libcephfs2-2:20.2.0-712.g70f8415b.el9.x86_64 30/140 2026-03-20T17:59:59.669 INFO:teuthology.orchestra.run.vm01.stdout:(33/138): ceph-prometheus-alerts-20.2.0-712.g70 154 kB/s | 17 kB 00:00 2026-03-20T17:59:59.678 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-cephfs-2:20.2.0-712.g70f8415b.el9.x86_64 31/140 2026-03-20T17:59:59.686 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 32/140 2026-03-20T17:59:59.715 INFO:teuthology.orchestra.run.vm03.stdout: Installing : zip-3.0-35.el9.x86_64 33/140 2026-03-20T17:59:59.721 INFO:teuthology.orchestra.run.vm03.stdout: Installing : luarocks-3.9.2-5.el9.noarch 34/140 2026-03-20T17:59:59.729 INFO:teuthology.orchestra.run.vm03.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 35/140 2026-03-20T17:59:59.784 INFO:teuthology.orchestra.run.vm01.stdout:(34/138): ceph-volume-20.2.0-712.g70f8415b.el9. 2.5 MB/s | 298 kB 00:00 2026-03-20T17:59:59.790 INFO:teuthology.orchestra.run.vm03.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 36/140 2026-03-20T17:59:59.807 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 37/140 2026-03-20T17:59:59.825 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/140 2026-03-20T17:59:59.832 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-rbd-2:20.2.0-712.g70f8415b.el9.x86_64 39/140 2026-03-20T17:59:59.841 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 40/140 2026-03-20T17:59:59.847 INFO:teuthology.orchestra.run.vm03.stdout: Installing : librados-devel-2:20.2.0-712.g70f8415b.el9.x86_64 41/140 2026-03-20T17:59:59.852 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 42/140 2026-03-20T17:59:59.870 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 43/140 2026-03-20T17:59:59.877 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 44/140 2026-03-20T17:59:59.884 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 45/140 2026-03-20T17:59:59.899 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 46/140 2026-03-20T17:59:59.911 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 47/140 2026-03-20T17:59:59.917 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 48/140 2026-03-20T17:59:59.929 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 49/140 2026-03-20T17:59:59.979 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 50/140 2026-03-20T18:00:00.124 INFO:teuthology.orchestra.run.vm01.stdout:(35/138): cephadm-20.2.0-712.g70f8415b.el9.noar 2.9 MB/s | 1.0 MB 00:00 2026-03-20T18:00:00.288 INFO:teuthology.orchestra.run.vm01.stdout:(36/138): ceph-mgr-diskprediction-local-20.2.0- 7.4 MB/s | 7.4 MB 00:01 2026-03-20T18:00:00.390 INFO:teuthology.orchestra.run.vm01.stdout:(37/138): bzip2-1.0.8-11.el9.x86_64.rpm 206 kB/s | 55 kB 00:00 2026-03-20T18:00:00.397 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 51/140 2026-03-20T18:00:00.414 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 52/140 2026-03-20T18:00:00.419 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 53/140 2026-03-20T18:00:00.427 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 54/140 2026-03-20T18:00:00.432 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 55/140 2026-03-20T18:00:00.443 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 56/140 2026-03-20T18:00:00.446 INFO:teuthology.orchestra.run.vm03.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 57/140 2026-03-20T18:00:00.449 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 58/140 2026-03-20T18:00:00.487 INFO:teuthology.orchestra.run.vm03.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 59/140 2026-03-20T18:00:00.508 INFO:teuthology.orchestra.run.vm01.stdout:(38/138): fuse-2.9.9-17.el9.x86_64.rpm 682 kB/s | 80 kB 00:00 2026-03-20T18:00:00.544 INFO:teuthology.orchestra.run.vm03.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 60/140 2026-03-20T18:00:00.557 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 61/140 2026-03-20T18:00:00.566 INFO:teuthology.orchestra.run.vm03.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 62/140 2026-03-20T18:00:00.572 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 63/140 2026-03-20T18:00:00.573 INFO:teuthology.orchestra.run.vm01.stdout:(39/138): ledmon-libs-1.1.0-3.el9.x86_64.rpm 620 kB/s | 40 kB 00:00 2026-03-20T18:00:00.581 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 64/140 2026-03-20T18:00:00.586 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 65/140 2026-03-20T18:00:00.596 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 66/140 2026-03-20T18:00:00.602 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 67/140 2026-03-20T18:00:00.640 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 68/140 2026-03-20T18:00:00.642 INFO:teuthology.orchestra.run.vm01.stdout:(40/138): libconfig-1.7.2-9.el9.x86_64.rpm 1.0 MB/s | 72 kB 00:00 2026-03-20T18:00:00.656 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 69/140 2026-03-20T18:00:00.665 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 70/140 2026-03-20T18:00:00.675 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 71/140 2026-03-20T18:00:00.684 INFO:teuthology.orchestra.run.vm01.stdout:(41/138): cryptsetup-2.8.1-3.el9.x86_64.rpm 888 kB/s | 351 kB 00:00 2026-03-20T18:00:00.729 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 72/140 2026-03-20T18:00:00.808 INFO:teuthology.orchestra.run.vm01.stdout:(42/138): libquadmath-11.5.0-14.el9.x86_64.rpm 1.5 MB/s | 184 kB 00:00 2026-03-20T18:00:00.871 INFO:teuthology.orchestra.run.vm01.stdout:(43/138): mailcap-2.1.49-5.el9.noarch.rpm 534 kB/s | 33 kB 00:00 2026-03-20T18:00:00.880 INFO:teuthology.orchestra.run.vm01.stdout:(44/138): libgfortran-11.5.0-14.el9.x86_64.rpm 3.3 MB/s | 794 kB 00:00 2026-03-20T18:00:00.946 INFO:teuthology.orchestra.run.vm01.stdout:(45/138): pciutils-3.7.0-7.el9.x86_64.rpm 1.2 MB/s | 93 kB 00:00 2026-03-20T18:00:00.963 INFO:teuthology.orchestra.run.vm01.stdout:(46/138): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2026-03-20T18:00:01.004 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 73/140 2026-03-20T18:00:01.036 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 74/140 2026-03-20T18:00:01.040 INFO:teuthology.orchestra.run.vm01.stdout:(47/138): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2026-03-20T18:00:01.041 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 75/140 2026-03-20T18:00:01.045 INFO:teuthology.orchestra.run.vm03.stdout: Installing : perl-Benchmark-1.23-483.el9.noarch 76/140 2026-03-20T18:00:01.113 INFO:teuthology.orchestra.run.vm03.stdout: Installing : openblas-0.3.29-1.el9.x86_64 77/140 2026-03-20T18:00:01.116 INFO:teuthology.orchestra.run.vm01.stdout:(48/138): python3-pycparser-2.20-6.el9.noarch.r 1.7 MB/s | 135 kB 00:00 2026-03-20T18:00:01.118 INFO:teuthology.orchestra.run.vm03.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 78/140 2026-03-20T18:00:01.144 INFO:teuthology.orchestra.run.vm01.stdout:(49/138): ceph-mgr-dashboard-20.2.0-712.g70f841 4.4 MB/s | 11 MB 00:02 2026-03-20T18:00:01.145 INFO:teuthology.orchestra.run.vm03.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 79/140 2026-03-20T18:00:01.150 INFO:teuthology.orchestra.run.vm01.stdout:(50/138): python3-cryptography-36.0.1-5.el9.x86 6.1 MB/s | 1.2 MB 00:00 2026-03-20T18:00:01.197 INFO:teuthology.orchestra.run.vm01.stdout:(51/138): python3-pyparsing-2.4.7-9.el9.noarch. 1.8 MB/s | 150 kB 00:00 2026-03-20T18:00:01.273 INFO:teuthology.orchestra.run.vm01.stdout:(52/138): python3-urllib3-1.26.5-7.el9.noarch.r 1.7 MB/s | 218 kB 00:00 2026-03-20T18:00:01.277 INFO:teuthology.orchestra.run.vm01.stdout:(53/138): unzip-6.0-59.el9.x86_64.rpm 2.2 MB/s | 182 kB 00:00 2026-03-20T18:00:01.354 INFO:teuthology.orchestra.run.vm01.stdout:(54/138): zip-3.0-35.el9.x86_64.rpm 3.2 MB/s | 266 kB 00:00 2026-03-20T18:00:01.430 INFO:teuthology.orchestra.run.vm01.stdout:(55/138): python3-requests-2.25.1-10.el9.noarch 442 kB/s | 126 kB 00:00 2026-03-20T18:00:01.496 INFO:teuthology.orchestra.run.vm01.stdout:(56/138): flexiblas-3.0.4-9.el9.x86_64.rpm 209 kB/s | 30 kB 00:00 2026-03-20T18:00:01.545 INFO:teuthology.orchestra.run.vm01.stdout:(57/138): flexiblas-openblas-openmp-3.0.4-9.el9 303 kB/s | 15 kB 00:00 2026-03-20T18:00:01.547 INFO:teuthology.orchestra.run.vm03.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 80/140 2026-03-20T18:00:01.550 INFO:teuthology.orchestra.run.vm01.stdout:(58/138): boost-program-options-1.75.0-13.el9.x 381 kB/s | 104 kB 00:00 2026-03-20T18:00:01.644 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 81/140 2026-03-20T18:00:01.647 INFO:teuthology.orchestra.run.vm01.stdout:(59/138): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2026-03-20T18:00:01.687 INFO:teuthology.orchestra.run.vm01.stdout:(60/138): libnbd-1.20.3-4.el9.x86_64.rpm 1.1 MB/s | 164 kB 00:00 2026-03-20T18:00:01.697 INFO:teuthology.orchestra.run.vm01.stdout:(61/138): librabbitmq-0.11.0-7.el9.x86_64.rpm 895 kB/s | 45 kB 00:00 2026-03-20T18:00:01.750 INFO:teuthology.orchestra.run.vm01.stdout:(62/138): libstoragemgmt-1.10.1-1.el9.x86_64.rp 4.6 MB/s | 246 kB 00:00 2026-03-20T18:00:01.802 INFO:teuthology.orchestra.run.vm01.stdout:(63/138): libxslt-1.1.34-12.el9.x86_64.rpm 4.4 MB/s | 233 kB 00:00 2026-03-20T18:00:01.831 INFO:teuthology.orchestra.run.vm01.stdout:(64/138): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2026-03-20T18:00:01.883 INFO:teuthology.orchestra.run.vm01.stdout:(65/138): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 188 kB 00:00 2026-03-20T18:00:01.933 INFO:teuthology.orchestra.run.vm01.stdout:(66/138): openblas-0.3.29-1.el9.x86_64.rpm 843 kB/s | 42 kB 00:00 2026-03-20T18:00:01.962 INFO:teuthology.orchestra.run.vm01.stdout:(67/138): flexiblas-netlib-3.0.4-9.el9.x86_64.r 5.6 MB/s | 3.0 MB 00:00 2026-03-20T18:00:02.012 INFO:teuthology.orchestra.run.vm01.stdout:(68/138): perl-Benchmark-1.23-483.el9.noarch.rp 536 kB/s | 26 kB 00:00 2026-03-20T18:00:02.068 INFO:teuthology.orchestra.run.vm01.stdout:(69/138): perl-Test-Harness-3.42-461.el9.noarch 5.2 MB/s | 295 kB 00:00 2026-03-20T18:00:02.129 INFO:teuthology.orchestra.run.vm01.stdout:(70/138): protobuf-3.14.0-17.el9.x86_64.rpm 16 MB/s | 1.0 MB 00:00 2026-03-20T18:00:02.178 INFO:teuthology.orchestra.run.vm01.stdout:(71/138): openblas-openmp-0.3.29-1.el9.x86_64.r 22 MB/s | 5.3 MB 00:00 2026-03-20T18:00:02.250 INFO:teuthology.orchestra.run.vm01.stdout:(72/138): lttng-ust-2.12.0-6.el9.x86_64.rpm 653 kB/s | 292 kB 00:00 2026-03-20T18:00:02.260 INFO:teuthology.orchestra.run.vm01.stdout:(73/138): python3-devel-3.9.25-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2026-03-20T18:00:02.302 INFO:teuthology.orchestra.run.vm01.stdout:(74/138): python3-jinja2-2.11.3-8.el9.noarch.rp 4.7 MB/s | 249 kB 00:00 2026-03-20T18:00:02.311 INFO:teuthology.orchestra.run.vm01.stdout:(75/138): python3-jmespath-1.0.1-1.el9.noarch.r 934 kB/s | 48 kB 00:00 2026-03-20T18:00:02.354 INFO:teuthology.orchestra.run.vm01.stdout:(76/138): python3-libstoragemgmt-1.10.1-1.el9.x 3.3 MB/s | 177 kB 00:00 2026-03-20T18:00:02.362 INFO:teuthology.orchestra.run.vm01.stdout:(77/138): python3-markupsafe-1.1.1-12.el9.x86_6 689 kB/s | 35 kB 00:00 2026-03-20T18:00:02.388 INFO:teuthology.orchestra.run.vm01.stdout:(78/138): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 6.0 MB 00:00 2026-03-20T18:00:02.422 INFO:teuthology.orchestra.run.vm01.stdout:(79/138): python3-numpy-f2py-1.23.5-2.el9.x86_6 7.2 MB/s | 442 kB 00:00 2026-03-20T18:00:02.438 INFO:teuthology.orchestra.run.vm01.stdout:(80/138): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2026-03-20T18:00:02.474 INFO:teuthology.orchestra.run.vm01.stdout:(81/138): python3-protobuf-3.14.0-17.el9.noarch 5.1 MB/s | 267 kB 00:00 2026-03-20T18:00:02.490 INFO:teuthology.orchestra.run.vm01.stdout:(82/138): python3-pyasn1-0.4.8-7.el9.noarch.rpm 3.0 MB/s | 157 kB 00:00 2026-03-20T18:00:02.505 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 82/140 2026-03-20T18:00:02.526 INFO:teuthology.orchestra.run.vm01.stdout:(83/138): python3-pyasn1-modules-0.4.8-7.el9.no 5.2 MB/s | 277 kB 00:00 2026-03-20T18:00:02.532 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 83/140 2026-03-20T18:00:02.539 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 84/140 2026-03-20T18:00:02.541 INFO:teuthology.orchestra.run.vm01.stdout:(84/138): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2026-03-20T18:00:02.542 INFO:teuthology.orchestra.run.vm03.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 85/140 2026-03-20T18:00:02.549 INFO:teuthology.orchestra.run.vm03.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 86/140 2026-03-20T18:00:02.592 INFO:teuthology.orchestra.run.vm01.stdout:(85/138): python3-toml-0.10.2-6.el9.noarch.rpm 815 kB/s | 42 kB 00:00 2026-03-20T18:00:02.651 INFO:teuthology.orchestra.run.vm01.stdout:(86/138): qatlib-25.08.0-2.el9.x86_64.rpm 4.0 MB/s | 240 kB 00:00 2026-03-20T18:00:02.700 INFO:teuthology.orchestra.run.vm01.stdout:(87/138): qatlib-service-25.08.0-2.el9.x86_64.r 759 kB/s | 37 kB 00:00 2026-03-20T18:00:02.751 INFO:teuthology.orchestra.run.vm01.stdout:(88/138): qatzip-libs-1.3.1-1.el9.x86_64.rpm 1.3 MB/s | 66 kB 00:00 2026-03-20T18:00:02.802 INFO:teuthology.orchestra.run.vm01.stdout:(89/138): socat-1.7.4.1-8.el9.x86_64.rpm 5.9 MB/s | 303 kB 00:00 2026-03-20T18:00:02.852 INFO:teuthology.orchestra.run.vm01.stdout:(90/138): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2026-03-20T18:00:02.871 INFO:teuthology.orchestra.run.vm03.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 87/140 2026-03-20T18:00:02.873 INFO:teuthology.orchestra.run.vm03.stdout: Installing : librgw2-2:20.2.0-712.g70f8415b.el9.x86_64 88/140 2026-03-20T18:00:02.896 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: librgw2-2:20.2.0-712.g70f8415b.el9.x86_64 88/140 2026-03-20T18:00:02.897 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-rgw-2:20.2.0-712.g70f8415b.el9.x86_64 89/140 2026-03-20T18:00:02.998 INFO:teuthology.orchestra.run.vm01.stdout:(91/138): lua-devel-5.4.4-4.el9.x86_64.rpm 153 kB/s | 22 kB 00:00 2026-03-20T18:00:03.074 INFO:teuthology.orchestra.run.vm01.stdout:(92/138): python3-numpy-1.23.5-2.el9.x86_64.rpm 8.5 MB/s | 6.1 MB 00:00 2026-03-20T18:00:03.093 INFO:teuthology.orchestra.run.vm01.stdout:(93/138): abseil-cpp-20211102.0-4.el9.x86_64.rp 28 MB/s | 551 kB 00:00 2026-03-20T18:00:03.110 INFO:teuthology.orchestra.run.vm01.stdout:(94/138): gperftools-libs-2.9.1-3.el9.x86_64.rp 18 MB/s | 308 kB 00:00 2026-03-20T18:00:03.113 INFO:teuthology.orchestra.run.vm01.stdout:(95/138): grpc-data-1.46.7-10.el9.noarch.rpm 9.2 MB/s | 19 kB 00:00 2026-03-20T18:00:03.175 INFO:teuthology.orchestra.run.vm01.stdout:(96/138): python3-scipy-1.9.3-2.el9.x86_64.rpm 30 MB/s | 19 MB 00:00 2026-03-20T18:00:03.196 INFO:teuthology.orchestra.run.vm01.stdout:(97/138): libarrow-doc-9.0.0-15.el9.noarch.rpm 1.2 MB/s | 25 kB 00:00 2026-03-20T18:00:03.203 INFO:teuthology.orchestra.run.vm01.stdout:(98/138): liboath-2.6.12-1.el9.x86_64.rpm 7.0 MB/s | 49 kB 00:00 2026-03-20T18:00:03.208 INFO:teuthology.orchestra.run.vm01.stdout:(99/138): libunwind-1.6.2-1.el9.x86_64.rpm 13 MB/s | 67 kB 00:00 2026-03-20T18:00:03.223 INFO:teuthology.orchestra.run.vm01.stdout:(100/138): libarrow-9.0.0-15.el9.x86_64.rpm 40 MB/s | 4.4 MB 00:00 2026-03-20T18:00:03.225 INFO:teuthology.orchestra.run.vm01.stdout:(101/138): luarocks-3.9.2-5.el9.noarch.rpm 8.9 MB/s | 151 kB 00:00 2026-03-20T18:00:03.238 INFO:teuthology.orchestra.run.vm01.stdout:(102/138): parquet-libs-9.0.0-15.el9.x86_64.rpm 55 MB/s | 838 kB 00:00 2026-03-20T18:00:03.242 INFO:teuthology.orchestra.run.vm01.stdout:(103/138): python3-asyncssh-2.13.2-5.el9.noarch 31 MB/s | 548 kB 00:00 2026-03-20T18:00:03.244 INFO:teuthology.orchestra.run.vm01.stdout:(104/138): python3-autocommand-2.2.2-8.el9.noar 5.6 MB/s | 29 kB 00:00 2026-03-20T18:00:03.245 INFO:teuthology.orchestra.run.vm01.stdout:(105/138): python3-backports-tarfile-1.2.0-1.el 24 MB/s | 60 kB 00:00 2026-03-20T18:00:03.247 INFO:teuthology.orchestra.run.vm01.stdout:(106/138): python3-bcrypt-3.2.2-1.el9.x86_64.rp 12 MB/s | 43 kB 00:00 2026-03-20T18:00:03.248 INFO:teuthology.orchestra.run.vm01.stdout:(107/138): python3-cachetools-4.2.4-1.el9.noarc 9.2 MB/s | 32 kB 00:00 2026-03-20T18:00:03.250 INFO:teuthology.orchestra.run.vm01.stdout:(108/138): python3-certifi-2023.05.07-4.el9.noa 6.3 MB/s | 14 kB 00:00 2026-03-20T18:00:03.253 INFO:teuthology.orchestra.run.vm01.stdout:(109/138): python3-cheroot-10.0.1-4.el9.noarch. 39 MB/s | 173 kB 00:00 2026-03-20T18:00:03.257 INFO:teuthology.orchestra.run.vm01.stdout:(110/138): python3-cherrypy-18.6.1-2.el9.noarch 50 MB/s | 358 kB 00:00 2026-03-20T18:00:03.260 INFO:teuthology.orchestra.run.vm01.stdout:(111/138): python3-google-auth-2.45.0-1.el9.noa 34 MB/s | 254 kB 00:00 2026-03-20T18:00:03.266 INFO:teuthology.orchestra.run.vm01.stdout:(112/138): python3-grpcio-tools-1.46.7-10.el9.x 25 MB/s | 144 kB 00:00 2026-03-20T18:00:03.271 INFO:teuthology.orchestra.run.vm01.stdout:(113/138): python3-jaraco-8.2.1-3.el9.noarch.rp 2.4 MB/s | 11 kB 00:00 2026-03-20T18:00:03.278 INFO:teuthology.orchestra.run.vm01.stdout:(114/138): python3-jaraco-classes-3.2.1-5.el9.n 2.4 MB/s | 18 kB 00:00 2026-03-20T18:00:03.288 INFO:teuthology.orchestra.run.vm01.stdout:(115/138): python3-grpcio-1.46.7-10.el9.x86_64. 67 MB/s | 2.0 MB 00:00 2026-03-20T18:00:03.289 INFO:teuthology.orchestra.run.vm01.stdout:(116/138): python3-jaraco-collections-3.0.0-8.e 2.1 MB/s | 23 kB 00:00 2026-03-20T18:00:03.290 INFO:teuthology.orchestra.run.vm01.stdout:(117/138): python3-jaraco-context-6.0.1-3.el9.n 9.0 MB/s | 20 kB 00:00 2026-03-20T18:00:03.291 INFO:teuthology.orchestra.run.vm01.stdout:(118/138): python3-jaraco-functools-3.5.0-2.el9 9.9 MB/s | 19 kB 00:00 2026-03-20T18:00:03.292 INFO:teuthology.orchestra.run.vm01.stdout:(119/138): python3-jaraco-text-4.0.0-2.el9.noar 13 MB/s | 26 kB 00:00 2026-03-20T18:00:03.295 INFO:teuthology.orchestra.run.vm01.stdout:(120/138): python3-more-itertools-8.12.0-2.el9. 25 MB/s | 79 kB 00:00 2026-03-20T18:00:03.299 INFO:teuthology.orchestra.run.vm01.stdout:(121/138): python3-natsort-7.1.1-5.el9.noarch.r 17 MB/s | 58 kB 00:00 2026-03-20T18:00:03.301 INFO:teuthology.orchestra.run.vm01.stdout:(122/138): python3-portend-3.1.0-2.el9.noarch.r 7.5 MB/s | 16 kB 00:00 2026-03-20T18:00:03.307 INFO:teuthology.orchestra.run.vm01.stdout:(123/138): python3-pyOpenSSL-21.0.0-1.el9.noarc 15 MB/s | 90 kB 00:00 2026-03-20T18:00:03.312 INFO:teuthology.orchestra.run.vm01.stdout:(124/138): python3-repoze-lru-0.7-16.el9.noarch 5.9 MB/s | 31 kB 00:00 2026-03-20T18:00:03.319 INFO:teuthology.orchestra.run.vm01.stdout:(125/138): python3-routes-2.5.1-5.el9.noarch.rp 28 MB/s | 188 kB 00:00 2026-03-20T18:00:03.324 INFO:teuthology.orchestra.run.vm01.stdout:(126/138): python3-kubernetes-26.1.0-3.el9.noar 31 MB/s | 1.0 MB 00:00 2026-03-20T18:00:03.325 INFO:teuthology.orchestra.run.vm01.stdout:(127/138): python3-rsa-4.9-2.el9.noarch.rpm 10 MB/s | 59 kB 00:00 2026-03-20T18:00:03.327 INFO:teuthology.orchestra.run.vm01.stdout:(128/138): python3-tempora-5.0.0-2.el9.noarch.r 12 MB/s | 36 kB 00:00 2026-03-20T18:00:03.331 INFO:teuthology.orchestra.run.vm01.stdout:(129/138): python3-websocket-client-1.2.3-2.el9 25 MB/s | 90 kB 00:00 2026-03-20T18:00:03.333 INFO:teuthology.orchestra.run.vm01.stdout:(130/138): python3-xmltodict-0.12.0-15.el9.noar 9.0 MB/s | 22 kB 00:00 2026-03-20T18:00:03.335 INFO:teuthology.orchestra.run.vm01.stdout:(131/138): python3-zc-lockfile-2.0-10.el9.noarc 9.5 MB/s | 20 kB 00:00 2026-03-20T18:00:03.340 INFO:teuthology.orchestra.run.vm01.stdout:(132/138): python3-typing-extensions-4.15.0-1.e 5.8 MB/s | 86 kB 00:00 2026-03-20T18:00:03.343 INFO:teuthology.orchestra.run.vm01.stdout:(133/138): re2-20211101-20.el9.x86_64.rpm 26 MB/s | 191 kB 00:00 2026-03-20T18:00:03.360 INFO:teuthology.orchestra.run.vm01.stdout:(134/138): s3cmd-2.4.0-1.el9.noarch.rpm 10 MB/s | 206 kB 00:00 2026-03-20T18:00:03.384 INFO:teuthology.orchestra.run.vm01.stdout:(135/138): thrift-0.15.0-4.el9.x86_64.rpm 39 MB/s | 1.6 MB 00:00 2026-03-20T18:00:03.859 INFO:teuthology.orchestra.run.vm01.stdout:(136/138): protobuf-compiler-3.14.0-17.el9.x86_ 1.0 MB/s | 862 kB 00:00 2026-03-20T18:00:04.171 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-common-2:20.2.0-712.g70f8415b.el9.x86_64 90/140 2026-03-20T18:00:04.176 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-common-2:20.2.0-712.g70f8415b.el9.x86_64 90/140 2026-03-20T18:00:04.198 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-common-2:20.2.0-712.g70f8415b.el9.x86_64 90/140 2026-03-20T18:00:04.213 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 91/140 2026-03-20T18:00:04.223 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-packaging-20.9-5.el9.noarch 92/140 2026-03-20T18:00:04.242 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-ply-3.11-14.el9.noarch 93/140 2026-03-20T18:00:04.265 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 94/140 2026-03-20T18:00:04.367 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 95/140 2026-03-20T18:00:04.381 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 96/140 2026-03-20T18:00:04.415 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 97/140 2026-03-20T18:00:04.456 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 98/140 2026-03-20T18:00:04.533 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 99/140 2026-03-20T18:00:04.602 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 100/140 2026-03-20T18:00:04.611 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 101/140 2026-03-20T18:00:04.620 INFO:teuthology.orchestra.run.vm03.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 102/140 2026-03-20T18:00:04.625 INFO:teuthology.orchestra.run.vm03.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 103/140 2026-03-20T18:00:04.628 INFO:teuthology.orchestra.run.vm03.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 104/140 2026-03-20T18:00:04.647 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 104/140 2026-03-20T18:00:04.985 INFO:teuthology.orchestra.run.vm03.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 105/140 2026-03-20T18:00:04.992 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-base-2:20.2.0-712.g70f8415b.el9.x86_64 106/140 2026-03-20T18:00:05.038 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-base-2:20.2.0-712.g70f8415b.el9.x86_64 106/140 2026-03-20T18:00:05.038 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-03-20T18:00:05.038 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-03-20T18:00:05.038 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:05.060 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-selinux-2:20.2.0-712.g70f8415b.el9.x86_64 107/140 2026-03-20T18:00:05.078 INFO:teuthology.orchestra.run.vm01.stdout:(137/138): librbd1-20.2.0-712.g70f8415b.el9.x86 1.7 MB/s | 2.8 MB 00:01 2026-03-20T18:00:05.343 INFO:teuthology.orchestra.run.vm01.stdout:(138/138): librados2-20.2.0-712.g70f8415b.el9.x 1.8 MB/s | 3.5 MB 00:01 2026-03-20T18:00:05.345 INFO:teuthology.orchestra.run.vm01.stdout:-------------------------------------------------------------------------------- 2026-03-20T18:00:05.345 INFO:teuthology.orchestra.run.vm01.stdout:Total 13 MB/s | 267 MB 00:20 2026-03-20T18:00:05.936 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction check 2026-03-20T18:00:05.995 INFO:teuthology.orchestra.run.vm01.stdout:Transaction check succeeded. 2026-03-20T18:00:05.995 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction test 2026-03-20T18:00:07.022 INFO:teuthology.orchestra.run.vm01.stdout:Transaction test succeeded. 2026-03-20T18:00:07.022 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction 2026-03-20T18:00:08.136 INFO:teuthology.orchestra.run.vm01.stdout: Preparing : 1/1 2026-03-20T18:00:08.145 INFO:teuthology.orchestra.run.vm01.stdout: Installing : thrift-0.15.0-4.el9.x86_64 1/140 2026-03-20T18:00:08.148 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/140 2026-03-20T18:00:08.159 INFO:teuthology.orchestra.run.vm01.stdout: Installing : liboath-2.6.12-1.el9.x86_64 3/140 2026-03-20T18:00:08.329 INFO:teuthology.orchestra.run.vm01.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/140 2026-03-20T18:00:08.331 INFO:teuthology.orchestra.run.vm01.stdout: Upgrading : librados2-2:20.2.0-712.g70f8415b.el9.x86_64 5/140 2026-03-20T18:00:08.367 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: librados2-2:20.2.0-712.g70f8415b.el9.x86_64 5/140 2026-03-20T18:00:08.378 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-rados-2:20.2.0-712.g70f8415b.el9.x86_64 6/140 2026-03-20T18:00:08.383 INFO:teuthology.orchestra.run.vm01.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 7/140 2026-03-20T18:00:08.387 INFO:teuthology.orchestra.run.vm01.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 8/140 2026-03-20T18:00:08.392 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 9/140 2026-03-20T18:00:08.397 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 10/140 2026-03-20T18:00:08.544 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 11/140 2026-03-20T18:00:08.587 INFO:teuthology.orchestra.run.vm01.stdout: Upgrading : librbd1-2:20.2.0-712.g70f8415b.el9.x86_64 12/140 2026-03-20T18:00:08.610 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: librbd1-2:20.2.0-712.g70f8415b.el9.x86_64 12/140 2026-03-20T18:00:08.612 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libcephsqlite-2:20.2.0-712.g70f8415b.el9.x86_64 13/140 2026-03-20T18:00:08.637 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: libcephsqlite-2:20.2.0-712.g70f8415b.el9.x86_64 13/140 2026-03-20T18:00:08.639 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libradosstriper1-2:20.2.0-712.g70f8415b.el9.x86_ 14/140 2026-03-20T18:00:08.655 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: libradosstriper1-2:20.2.0-712.g70f8415b.el9.x86_ 14/140 2026-03-20T18:00:08.691 INFO:teuthology.orchestra.run.vm01.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/140 2026-03-20T18:00:08.720 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/140 2026-03-20T18:00:08.732 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 17/140 2026-03-20T18:00:08.739 INFO:teuthology.orchestra.run.vm01.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 18/140 2026-03-20T18:00:08.743 INFO:teuthology.orchestra.run.vm01.stdout: Installing : lua-5.4.4-4.el9.x86_64 19/140 2026-03-20T18:00:08.749 INFO:teuthology.orchestra.run.vm01.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 20/140 2026-03-20T18:00:08.778 INFO:teuthology.orchestra.run.vm01.stdout: Installing : unzip-6.0-59.el9.x86_64 21/140 2026-03-20T18:00:08.796 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 22/140 2026-03-20T18:00:08.801 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 23/140 2026-03-20T18:00:08.810 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 24/140 2026-03-20T18:00:08.812 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 25/140 2026-03-20T18:00:08.854 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 26/140 2026-03-20T18:00:08.863 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-ceph-common-2:20.2.0-712.g70f8415b.el9.x 27/140 2026-03-20T18:00:08.865 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-ceph-argparse-2:20.2.0-712.g70f8415b.el9 28/140 2026-03-20T18:00:08.866 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libcephfs-proxy2-2:20.2.0-712.g70f8415b.el9.x86_ 29/140 2026-03-20T18:00:08.921 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: libcephfs-proxy2-2:20.2.0-712.g70f8415b.el9.x86_ 29/140 2026-03-20T18:00:08.923 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libcephfs2-2:20.2.0-712.g70f8415b.el9.x86_64 30/140 2026-03-20T18:00:08.983 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: libcephfs2-2:20.2.0-712.g70f8415b.el9.x86_64 30/140 2026-03-20T18:00:09.000 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-cephfs-2:20.2.0-712.g70f8415b.el9.x86_64 31/140 2026-03-20T18:00:09.010 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 32/140 2026-03-20T18:00:09.042 INFO:teuthology.orchestra.run.vm01.stdout: Installing : zip-3.0-35.el9.x86_64 33/140 2026-03-20T18:00:09.047 INFO:teuthology.orchestra.run.vm01.stdout: Installing : luarocks-3.9.2-5.el9.noarch 34/140 2026-03-20T18:00:09.056 INFO:teuthology.orchestra.run.vm01.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 35/140 2026-03-20T18:00:09.116 INFO:teuthology.orchestra.run.vm01.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 36/140 2026-03-20T18:00:09.134 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 37/140 2026-03-20T18:00:09.155 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/140 2026-03-20T18:00:09.167 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-rbd-2:20.2.0-712.g70f8415b.el9.x86_64 39/140 2026-03-20T18:00:09.176 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 40/140 2026-03-20T18:00:09.184 INFO:teuthology.orchestra.run.vm01.stdout: Installing : librados-devel-2:20.2.0-712.g70f8415b.el9.x86_64 41/140 2026-03-20T18:00:09.190 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 42/140 2026-03-20T18:00:09.210 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 43/140 2026-03-20T18:00:09.217 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 44/140 2026-03-20T18:00:09.224 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 45/140 2026-03-20T18:00:09.240 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 46/140 2026-03-20T18:00:09.253 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 47/140 2026-03-20T18:00:09.259 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 48/140 2026-03-20T18:00:09.269 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 49/140 2026-03-20T18:00:09.316 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 50/140 2026-03-20T18:00:09.702 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 51/140 2026-03-20T18:00:09.718 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 52/140 2026-03-20T18:00:09.722 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 53/140 2026-03-20T18:00:09.730 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 54/140 2026-03-20T18:00:09.733 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 55/140 2026-03-20T18:00:09.745 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 56/140 2026-03-20T18:00:09.749 INFO:teuthology.orchestra.run.vm01.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 57/140 2026-03-20T18:00:09.752 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 58/140 2026-03-20T18:00:09.794 INFO:teuthology.orchestra.run.vm01.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 59/140 2026-03-20T18:00:09.848 INFO:teuthology.orchestra.run.vm01.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 60/140 2026-03-20T18:00:09.863 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 61/140 2026-03-20T18:00:09.871 INFO:teuthology.orchestra.run.vm01.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 62/140 2026-03-20T18:00:09.893 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 63/140 2026-03-20T18:00:09.902 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 64/140 2026-03-20T18:00:09.907 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 65/140 2026-03-20T18:00:09.917 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 66/140 2026-03-20T18:00:09.923 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 67/140 2026-03-20T18:00:09.960 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 68/140 2026-03-20T18:00:09.974 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 69/140 2026-03-20T18:00:09.983 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 70/140 2026-03-20T18:00:09.995 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 71/140 2026-03-20T18:00:10.041 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 72/140 2026-03-20T18:00:10.341 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 73/140 2026-03-20T18:00:10.377 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 74/140 2026-03-20T18:00:10.403 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 75/140 2026-03-20T18:00:10.407 INFO:teuthology.orchestra.run.vm01.stdout: Installing : perl-Benchmark-1.23-483.el9.noarch 76/140 2026-03-20T18:00:10.482 INFO:teuthology.orchestra.run.vm01.stdout: Installing : openblas-0.3.29-1.el9.x86_64 77/140 2026-03-20T18:00:10.554 INFO:teuthology.orchestra.run.vm01.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 78/140 2026-03-20T18:00:10.604 INFO:teuthology.orchestra.run.vm01.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 79/140 2026-03-20T18:00:11.040 INFO:teuthology.orchestra.run.vm01.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 80/140 2026-03-20T18:00:11.131 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 81/140 2026-03-20T18:00:11.493 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-selinux-2:20.2.0-712.g70f8415b.el9.x86_64 107/140 2026-03-20T18:00:11.493 INFO:teuthology.orchestra.run.vm03.stdout:skipping the directory /sys 2026-03-20T18:00:11.493 INFO:teuthology.orchestra.run.vm03.stdout:skipping the directory /proc 2026-03-20T18:00:11.493 INFO:teuthology.orchestra.run.vm03.stdout:skipping the directory /mnt 2026-03-20T18:00:11.493 INFO:teuthology.orchestra.run.vm03.stdout:skipping the directory /var/tmp 2026-03-20T18:00:11.493 INFO:teuthology.orchestra.run.vm03.stdout:skipping the directory /home 2026-03-20T18:00:11.493 INFO:teuthology.orchestra.run.vm03.stdout:skipping the directory /root 2026-03-20T18:00:11.493 INFO:teuthology.orchestra.run.vm03.stdout:skipping the directory /tmp 2026-03-20T18:00:11.493 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:11.614 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-mds-2:20.2.0-712.g70f8415b.el9.x86_64 108/140 2026-03-20T18:00:11.638 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-mds-2:20.2.0-712.g70f8415b.el9.x86_64 108/140 2026-03-20T18:00:11.638 INFO:teuthology.orchestra.run.vm03.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:11.639 INFO:teuthology.orchestra.run.vm03.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-03-20T18:00:11.639 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-03-20T18:00:11.639 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-03-20T18:00:11.639 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:11.894 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-mon-2:20.2.0-712.g70f8415b.el9.x86_64 109/140 2026-03-20T18:00:11.916 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-mon-2:20.2.0-712.g70f8415b.el9.x86_64 109/140 2026-03-20T18:00:11.916 INFO:teuthology.orchestra.run.vm03.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:11.916 INFO:teuthology.orchestra.run.vm03.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-03-20T18:00:11.916 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-03-20T18:00:11.916 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-03-20T18:00:11.917 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:11.925 INFO:teuthology.orchestra.run.vm03.stdout: Installing : mailcap-2.1.49-5.el9.noarch 110/140 2026-03-20T18:00:11.928 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 111/140 2026-03-20T18:00:11.938 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 82/140 2026-03-20T18:00:11.947 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 112/140 2026-03-20T18:00:11.947 INFO:teuthology.orchestra.run.vm03.stdout:Creating group 'qat' with GID 994. 2026-03-20T18:00:11.947 INFO:teuthology.orchestra.run.vm03.stdout:Creating group 'libstoragemgmt' with GID 993. 2026-03-20T18:00:11.947 INFO:teuthology.orchestra.run.vm03.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 993 and GID 993. 2026-03-20T18:00:11.947 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:11.958 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 112/140 2026-03-20T18:00:11.965 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 83/140 2026-03-20T18:00:11.971 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 84/140 2026-03-20T18:00:11.975 INFO:teuthology.orchestra.run.vm01.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 85/140 2026-03-20T18:00:11.982 INFO:teuthology.orchestra.run.vm01.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 86/140 2026-03-20T18:00:11.986 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 112/140 2026-03-20T18:00:11.986 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-03-20T18:00:11.986 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:12.007 INFO:teuthology.orchestra.run.vm03.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 113/140 2026-03-20T18:00:12.033 INFO:teuthology.orchestra.run.vm03.stdout: Installing : fuse-2.9.9-17.el9.x86_64 114/140 2026-03-20T18:00:12.107 INFO:teuthology.orchestra.run.vm03.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 115/140 2026-03-20T18:00:12.110 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-volume-2:20.2.0-712.g70f8415b.el9.noarch 116/140 2026-03-20T18:00:12.123 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-volume-2:20.2.0-712.g70f8415b.el9.noarch 116/140 2026-03-20T18:00:12.123 INFO:teuthology.orchestra.run.vm03.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:12.123 INFO:teuthology.orchestra.run.vm03.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-03-20T18:00:12.123 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:12.305 INFO:teuthology.orchestra.run.vm01.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 87/140 2026-03-20T18:00:12.308 INFO:teuthology.orchestra.run.vm01.stdout: Installing : librgw2-2:20.2.0-712.g70f8415b.el9.x86_64 88/140 2026-03-20T18:00:12.330 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: librgw2-2:20.2.0-712.g70f8415b.el9.x86_64 88/140 2026-03-20T18:00:12.332 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-rgw-2:20.2.0-712.g70f8415b.el9.x86_64 89/140 2026-03-20T18:00:12.907 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-osd-2:20.2.0-712.g70f8415b.el9.x86_64 117/140 2026-03-20T18:00:12.930 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-osd-2:20.2.0-712.g70f8415b.el9.x86_64 117/140 2026-03-20T18:00:12.930 INFO:teuthology.orchestra.run.vm03.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:12.930 INFO:teuthology.orchestra.run.vm03.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-03-20T18:00:12.930 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-03-20T18:00:12.930 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-03-20T18:00:12.930 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:13.004 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: cephadm-2:20.2.0-712.g70f8415b.el9.noarch 118/140 2026-03-20T18:00:13.007 INFO:teuthology.orchestra.run.vm03.stdout: Installing : cephadm-2:20.2.0-712.g70f8415b.el9.noarch 118/140 2026-03-20T18:00:13.016 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-prometheus-alerts-2:20.2.0-712.g70f8415b.el 119/140 2026-03-20T18:00:13.045 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-grafana-dashboards-2:20.2.0-712.g70f8415b.e 120/140 2026-03-20T18:00:13.049 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-mgr-cephadm-2:20.2.0-712.g70f8415b.el9.noar 121/140 2026-03-20T18:00:13.569 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-common-2:20.2.0-712.g70f8415b.el9.x86_64 90/140 2026-03-20T18:00:13.599 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-common-2:20.2.0-712.g70f8415b.el9.x86_64 90/140 2026-03-20T18:00:13.620 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-common-2:20.2.0-712.g70f8415b.el9.x86_64 90/140 2026-03-20T18:00:13.634 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 91/140 2026-03-20T18:00:13.643 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-packaging-20.9-5.el9.noarch 92/140 2026-03-20T18:00:13.662 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-ply-3.11-14.el9.noarch 93/140 2026-03-20T18:00:13.685 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 94/140 2026-03-20T18:00:13.777 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 95/140 2026-03-20T18:00:13.792 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 96/140 2026-03-20T18:00:13.822 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 97/140 2026-03-20T18:00:13.861 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 98/140 2026-03-20T18:00:14.001 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 99/140 2026-03-20T18:00:14.012 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 100/140 2026-03-20T18:00:14.019 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 101/140 2026-03-20T18:00:14.025 INFO:teuthology.orchestra.run.vm01.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 102/140 2026-03-20T18:00:14.030 INFO:teuthology.orchestra.run.vm01.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 103/140 2026-03-20T18:00:14.031 INFO:teuthology.orchestra.run.vm01.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 104/140 2026-03-20T18:00:14.050 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 104/140 2026-03-20T18:00:14.302 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.2.0-712.g70f8415b.el9.noar 121/140 2026-03-20T18:00:14.312 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-mgr-dashboard-2:20.2.0-712.g70f8415b.el9.no 122/140 2026-03-20T18:00:14.396 INFO:teuthology.orchestra.run.vm01.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 105/140 2026-03-20T18:00:14.403 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-base-2:20.2.0-712.g70f8415b.el9.x86_64 106/140 2026-03-20T18:00:14.446 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-base-2:20.2.0-712.g70f8415b.el9.x86_64 106/140 2026-03-20T18:00:14.447 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-03-20T18:00:14.447 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-03-20T18:00:14.447 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:14.452 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-selinux-2:20.2.0-712.g70f8415b.el9.x86_64 107/140 2026-03-20T18:00:14.866 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.2.0-712.g70f8415b.el9.no 122/140 2026-03-20T18:00:14.868 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-mgr-diskprediction-local-2:20.2.0-712.g70f8 123/140 2026-03-20T18:00:14.942 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.2.0-712.g70f8 123/140 2026-03-20T18:00:14.994 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-mgr-modules-core-2:20.2.0-712.g70f8415b.el9 124/140 2026-03-20T18:00:14.997 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-mgr-2:20.2.0-712.g70f8415b.el9.x86_64 125/140 2026-03-20T18:00:15.022 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-mgr-2:20.2.0-712.g70f8415b.el9.x86_64 125/140 2026-03-20T18:00:15.022 INFO:teuthology.orchestra.run.vm03.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:15.022 INFO:teuthology.orchestra.run.vm03.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-03-20T18:00:15.022 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-03-20T18:00:15.022 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-03-20T18:00:15.022 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:15.036 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-mgr-rook-2:20.2.0-712.g70f8415b.el9.noarch 126/140 2026-03-20T18:00:15.049 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-mgr-rook-2:20.2.0-712.g70f8415b.el9.noarch 126/140 2026-03-20T18:00:15.094 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-2:20.2.0-712.g70f8415b.el9.x86_64 127/140 2026-03-20T18:00:16.290 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-fuse-2:20.2.0-712.g70f8415b.el9.x86_64 128/140 2026-03-20T18:00:16.293 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-radosgw-2:20.2.0-712.g70f8415b.el9.x86_64 129/140 2026-03-20T18:00:16.316 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-radosgw-2:20.2.0-712.g70f8415b.el9.x86_64 129/140 2026-03-20T18:00:16.316 INFO:teuthology.orchestra.run.vm03.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:16.316 INFO:teuthology.orchestra.run.vm03.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-03-20T18:00:16.316 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-03-20T18:00:16.316 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-03-20T18:00:16.316 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:16.328 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-immutable-object-cache-2:20.2.0-712.g70f841 130/140 2026-03-20T18:00:16.350 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.2.0-712.g70f841 130/140 2026-03-20T18:00:16.350 INFO:teuthology.orchestra.run.vm03.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:16.350 INFO:teuthology.orchestra.run.vm03.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-03-20T18:00:16.350 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:16.490 INFO:teuthology.orchestra.run.vm03.stdout: Installing : rbd-mirror-2:20.2.0-712.g70f8415b.el9.x86_64 131/140 2026-03-20T18:00:16.511 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: rbd-mirror-2:20.2.0-712.g70f8415b.el9.x86_64 131/140 2026-03-20T18:00:16.511 INFO:teuthology.orchestra.run.vm03.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:16.511 INFO:teuthology.orchestra.run.vm03.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-03-20T18:00:16.511 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-03-20T18:00:16.511 INFO:teuthology.orchestra.run.vm03.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-03-20T18:00:16.511 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:20.659 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-selinux-2:20.2.0-712.g70f8415b.el9.x86_64 107/140 2026-03-20T18:00:20.659 INFO:teuthology.orchestra.run.vm01.stdout:skipping the directory /sys 2026-03-20T18:00:20.659 INFO:teuthology.orchestra.run.vm01.stdout:skipping the directory /proc 2026-03-20T18:00:20.659 INFO:teuthology.orchestra.run.vm01.stdout:skipping the directory /mnt 2026-03-20T18:00:20.659 INFO:teuthology.orchestra.run.vm01.stdout:skipping the directory /var/tmp 2026-03-20T18:00:20.659 INFO:teuthology.orchestra.run.vm01.stdout:skipping the directory /home 2026-03-20T18:00:20.659 INFO:teuthology.orchestra.run.vm01.stdout:skipping the directory /root 2026-03-20T18:00:20.659 INFO:teuthology.orchestra.run.vm01.stdout:skipping the directory /tmp 2026-03-20T18:00:20.659 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:20.794 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-mds-2:20.2.0-712.g70f8415b.el9.x86_64 108/140 2026-03-20T18:00:20.825 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-mds-2:20.2.0-712.g70f8415b.el9.x86_64 108/140 2026-03-20T18:00:20.825 INFO:teuthology.orchestra.run.vm01.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:20.825 INFO:teuthology.orchestra.run.vm01.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-03-20T18:00:20.825 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-03-20T18:00:20.825 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-03-20T18:00:20.825 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:20.920 INFO:teuthology.orchestra.run.vm03.stdout: Installing : ceph-test-2:20.2.0-712.g70f8415b.el9.x86_64 132/140 2026-03-20T18:00:20.928 INFO:teuthology.orchestra.run.vm03.stdout: Installing : perl-Test-Harness-1:3.42-461.el9.noarch 133/140 2026-03-20T18:00:20.935 INFO:teuthology.orchestra.run.vm03.stdout: Installing : libcephfs-devel-2:20.2.0-712.g70f8415b.el9.x86_6 134/140 2026-03-20T18:00:20.947 INFO:teuthology.orchestra.run.vm03.stdout: Installing : rbd-fuse-2:20.2.0-712.g70f8415b.el9.x86_64 135/140 2026-03-20T18:00:20.967 INFO:teuthology.orchestra.run.vm03.stdout: Installing : rbd-nbd-2:20.2.0-712.g70f8415b.el9.x86_64 136/140 2026-03-20T18:00:20.978 INFO:teuthology.orchestra.run.vm03.stdout: Installing : s3cmd-2.4.0-1.el9.noarch 137/140 2026-03-20T18:00:20.982 INFO:teuthology.orchestra.run.vm03.stdout: Installing : bzip2-1.0.8-11.el9.x86_64 138/140 2026-03-20T18:00:20.983 INFO:teuthology.orchestra.run.vm03.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 139/140 2026-03-20T18:00:20.999 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 139/140 2026-03-20T18:00:21.000 INFO:teuthology.orchestra.run.vm03.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 140/140 2026-03-20T18:00:21.103 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-mon-2:20.2.0-712.g70f8415b.el9.x86_64 109/140 2026-03-20T18:00:21.125 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-mon-2:20.2.0-712.g70f8415b.el9.x86_64 109/140 2026-03-20T18:00:21.125 INFO:teuthology.orchestra.run.vm01.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:21.125 INFO:teuthology.orchestra.run.vm01.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-03-20T18:00:21.125 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-03-20T18:00:21.125 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-03-20T18:00:21.125 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:21.135 INFO:teuthology.orchestra.run.vm01.stdout: Installing : mailcap-2.1.49-5.el9.noarch 110/140 2026-03-20T18:00:21.137 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 111/140 2026-03-20T18:00:21.157 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 112/140 2026-03-20T18:00:21.158 INFO:teuthology.orchestra.run.vm01.stdout:Creating group 'qat' with GID 994. 2026-03-20T18:00:21.158 INFO:teuthology.orchestra.run.vm01.stdout:Creating group 'libstoragemgmt' with GID 993. 2026-03-20T18:00:21.158 INFO:teuthology.orchestra.run.vm01.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 993 and GID 993. 2026-03-20T18:00:21.158 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:21.168 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 112/140 2026-03-20T18:00:21.198 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 112/140 2026-03-20T18:00:21.198 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-03-20T18:00:21.198 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:21.222 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 113/140 2026-03-20T18:00:21.252 INFO:teuthology.orchestra.run.vm01.stdout: Installing : fuse-2.9.9-17.el9.x86_64 114/140 2026-03-20T18:00:21.335 INFO:teuthology.orchestra.run.vm01.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 115/140 2026-03-20T18:00:21.340 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-volume-2:20.2.0-712.g70f8415b.el9.noarch 116/140 2026-03-20T18:00:21.353 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-volume-2:20.2.0-712.g70f8415b.el9.noarch 116/140 2026-03-20T18:00:21.353 INFO:teuthology.orchestra.run.vm01.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:21.353 INFO:teuthology.orchestra.run.vm01.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-03-20T18:00:21.353 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:22.211 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-osd-2:20.2.0-712.g70f8415b.el9.x86_64 117/140 2026-03-20T18:00:22.236 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-osd-2:20.2.0-712.g70f8415b.el9.x86_64 117/140 2026-03-20T18:00:22.236 INFO:teuthology.orchestra.run.vm01.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:22.236 INFO:teuthology.orchestra.run.vm01.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-03-20T18:00:22.236 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-03-20T18:00:22.236 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-03-20T18:00:22.236 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:22.469 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: cephadm-2:20.2.0-712.g70f8415b.el9.noarch 118/140 2026-03-20T18:00:22.482 INFO:teuthology.orchestra.run.vm01.stdout: Installing : cephadm-2:20.2.0-712.g70f8415b.el9.noarch 118/140 2026-03-20T18:00:22.495 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-prometheus-alerts-2:20.2.0-712.g70f8415b.el 119/140 2026-03-20T18:00:22.526 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-grafana-dashboards-2:20.2.0-712.g70f8415b.e 120/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 140/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-2:20.2.0-712.g70f8415b.el9.x86_64 1/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-base-2:20.2.0-712.g70f8415b.el9.x86_64 2/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-common-2:20.2.0-712.g70f8415b.el9.x86_64 3/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-fuse-2:20.2.0-712.g70f8415b.el9.x86_64 4/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-immutable-object-cache-2:20.2.0-712.g70f841 5/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-mds-2:20.2.0-712.g70f8415b.el9.x86_64 6/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-mgr-2:20.2.0-712.g70f8415b.el9.x86_64 7/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-mon-2:20.2.0-712.g70f8415b.el9.x86_64 8/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-osd-2:20.2.0-712.g70f8415b.el9.x86_64 9/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-radosgw-2:20.2.0-712.g70f8415b.el9.x86_64 10/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-selinux-2:20.2.0-712.g70f8415b.el9.x86_64 11/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-test-2:20.2.0-712.g70f8415b.el9.x86_64 12/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libcephfs-devel-2:20.2.0-712.g70f8415b.el9.x86_6 13/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libcephfs-proxy2-2:20.2.0-712.g70f8415b.el9.x86_ 14/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libcephfs2-2:20.2.0-712.g70f8415b.el9.x86_64 15/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libcephsqlite-2:20.2.0-712.g70f8415b.el9.x86_64 16/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : librados-devel-2:20.2.0-712.g70f8415b.el9.x86_64 17/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libradosstriper1-2:20.2.0-712.g70f8415b.el9.x86_ 18/140 2026-03-20T18:00:22.529 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : librgw2-2:20.2.0-712.g70f8415b.el9.x86_64 19/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-ceph-argparse-2:20.2.0-712.g70f8415b.el9 20/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-ceph-common-2:20.2.0-712.g70f8415b.el9.x 21/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-cephfs-2:20.2.0-712.g70f8415b.el9.x86_64 22/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-rados-2:20.2.0-712.g70f8415b.el9.x86_64 23/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-rbd-2:20.2.0-712.g70f8415b.el9.x86_64 24/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-rgw-2:20.2.0-712.g70f8415b.el9.x86_64 25/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : rbd-fuse-2:20.2.0-712.g70f8415b.el9.x86_64 26/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : rbd-mirror-2:20.2.0-712.g70f8415b.el9.x86_64 27/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : rbd-nbd-2:20.2.0-712.g70f8415b.el9.x86_64 28/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-grafana-dashboards-2:20.2.0-712.g70f8415b.e 29/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-mgr-cephadm-2:20.2.0-712.g70f8415b.el9.noar 30/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-mgr-dashboard-2:20.2.0-712.g70f8415b.el9.no 31/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.2.0-712.g70f8 32/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-mgr-modules-core-2:20.2.0-712.g70f8415b.el9 33/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-mgr-rook-2:20.2.0-712.g70f8415b.el9.noarch 34/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-prometheus-alerts-2:20.2.0-712.g70f8415b.el 35/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ceph-volume-2:20.2.0-712.g70f8415b.el9.noarch 36/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : cephadm-2:20.2.0-712.g70f8415b.el9.noarch 37/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : bzip2-1.0.8-11.el9.x86_64 38/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 39/140 2026-03-20T18:00:22.530 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : fuse-2.9.9-17.el9.x86_64 40/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 41/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 42/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 43/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 44/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 45/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 46/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 47/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 48/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-ply-3.11-14.el9.noarch 49/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 50/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 51/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 52/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 53/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : unzip-6.0-59.el9.x86_64 54/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : zip-3.0-35.el9.x86_64 55/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 56/140 2026-03-20T18:00:22.532 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 57/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 58/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 59/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 60/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 61/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 62/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 63/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 64/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 65/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 66/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : lua-5.4.4-4.el9.x86_64 67/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 68/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 69/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : perl-Benchmark-1.23-483.el9.noarch 70/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : perl-Test-Harness-1:3.42-461.el9.noarch 71/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 72/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 73/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 74/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 75/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 76/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 77/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 78/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 79/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 80/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 81/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 82/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 83/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 84/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 86/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 87/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 88/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 89/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 90/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 91/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 92/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 93/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 94/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 95/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 96/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 97/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 98/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 99/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 100/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 102/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 103/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 104/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 105/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 106/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 107/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 108/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 109/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 110/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 111/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 112/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 113/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 114/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 115/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 116/140 2026-03-20T18:00:22.533 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 117/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 118/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 119/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 120/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 121/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 122/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 123/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 124/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 125/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 126/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 127/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 128/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 129/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 130/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 132/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : re2-1:20211101-20.el9.x86_64 134/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : s3cmd-2.4.0-1.el9.noarch 135/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 136/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : librados2-2:20.2.0-712.g70f8415b.el9.x86_64 137/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 138/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : librbd1-2:20.2.0-712.g70f8415b.el9.x86_64 139/140 2026-03-20T18:00:22.534 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-mgr-cephadm-2:20.2.0-712.g70f8415b.el9.noar 121/140 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 140/140 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout:Upgraded: 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: librados2-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: librbd1-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout:Installed: 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: bzip2-1.0.8-11.el9.x86_64 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-base-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-common-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-fuse-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-grafana-dashboards-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-immutable-object-cache-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mds-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mgr-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mgr-cephadm-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mgr-dashboard-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mgr-diskprediction-local-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mgr-modules-core-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mgr-rook-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:22.652 INFO:teuthology.orchestra.run.vm03.stdout: ceph-mon-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: ceph-osd-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: ceph-prometheus-alerts-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: ceph-radosgw-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: ceph-selinux-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: ceph-test-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: ceph-volume-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: cephadm-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: fuse-2.9.9-17.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: grpc-data-1.46.7-10.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libarrow-9.0.0-15.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libcephfs-devel-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libcephfs-proxy2-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libcephfs2-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libcephsqlite-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libconfig-1.7.2-9.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libnbd-1.20.3-4.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: liboath-2.6.12-1.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: librados-devel-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libradosstriper1-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: librgw2-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libunwind-1.6.2-1.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: libxslt-1.1.34-12.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: lua-5.4.4-4.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: luarocks-3.9.2-5.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: mailcap-2.1.49-5.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: openblas-0.3.29-1.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: pciutils-3.7.0-7.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: perl-Benchmark-1.23-483.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: perl-Test-Harness-1:3.42-461.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: protobuf-3.14.0-17.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: python3-babel-2.9.1-2.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: python3-ceph-argparse-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: python3-ceph-common-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.653 INFO:teuthology.orchestra.run.vm03.stdout: python3-cephfs-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-packaging-20.9-5.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-ply-3.11-14.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-portend-3.1.0-2.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-pycparser-2.20-6.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-rados-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-rbd-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-requests-2.25.1-10.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-rgw-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-routes-2.5.1-5.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-rsa-4.9-2.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-toml-0.10.2-6.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: qatlib-25.08.0-2.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: rbd-fuse-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: rbd-mirror-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: rbd-nbd-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: re2-1:20211101-20.el9.x86_64 2026-03-20T18:00:22.654 INFO:teuthology.orchestra.run.vm03.stdout: s3cmd-2.4.0-1.el9.noarch 2026-03-20T18:00:22.655 INFO:teuthology.orchestra.run.vm03.stdout: socat-1.7.4.1-8.el9.x86_64 2026-03-20T18:00:22.655 INFO:teuthology.orchestra.run.vm03.stdout: thrift-0.15.0-4.el9.x86_64 2026-03-20T18:00:22.655 INFO:teuthology.orchestra.run.vm03.stdout: unzip-6.0-59.el9.x86_64 2026-03-20T18:00:22.655 INFO:teuthology.orchestra.run.vm03.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-03-20T18:00:22.655 INFO:teuthology.orchestra.run.vm03.stdout: zip-3.0-35.el9.x86_64 2026-03-20T18:00:22.655 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:22.655 INFO:teuthology.orchestra.run.vm03.stdout:Complete! 2026-03-20T18:00:22.749 DEBUG:teuthology.parallel:result is None 2026-03-20T18:00:23.859 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.2.0-712.g70f8415b.el9.noar 121/140 2026-03-20T18:00:23.870 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-mgr-dashboard-2:20.2.0-712.g70f8415b.el9.no 122/140 2026-03-20T18:00:24.403 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.2.0-712.g70f8415b.el9.no 122/140 2026-03-20T18:00:24.405 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-mgr-diskprediction-local-2:20.2.0-712.g70f8 123/140 2026-03-20T18:00:24.468 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.2.0-712.g70f8 123/140 2026-03-20T18:00:24.517 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-mgr-modules-core-2:20.2.0-712.g70f8415b.el9 124/140 2026-03-20T18:00:24.520 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-mgr-2:20.2.0-712.g70f8415b.el9.x86_64 125/140 2026-03-20T18:00:24.541 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-mgr-2:20.2.0-712.g70f8415b.el9.x86_64 125/140 2026-03-20T18:00:24.541 INFO:teuthology.orchestra.run.vm01.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:24.541 INFO:teuthology.orchestra.run.vm01.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-03-20T18:00:24.541 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-03-20T18:00:24.541 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-03-20T18:00:24.541 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:24.555 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-mgr-rook-2:20.2.0-712.g70f8415b.el9.noarch 126/140 2026-03-20T18:00:24.565 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-mgr-rook-2:20.2.0-712.g70f8415b.el9.noarch 126/140 2026-03-20T18:00:24.609 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-2:20.2.0-712.g70f8415b.el9.x86_64 127/140 2026-03-20T18:00:25.742 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-fuse-2:20.2.0-712.g70f8415b.el9.x86_64 128/140 2026-03-20T18:00:25.745 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-radosgw-2:20.2.0-712.g70f8415b.el9.x86_64 129/140 2026-03-20T18:00:25.765 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-radosgw-2:20.2.0-712.g70f8415b.el9.x86_64 129/140 2026-03-20T18:00:25.765 INFO:teuthology.orchestra.run.vm01.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:25.765 INFO:teuthology.orchestra.run.vm01.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-03-20T18:00:25.765 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-03-20T18:00:25.765 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-03-20T18:00:25.765 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:25.778 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-immutable-object-cache-2:20.2.0-712.g70f841 130/140 2026-03-20T18:00:25.798 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.2.0-712.g70f841 130/140 2026-03-20T18:00:25.798 INFO:teuthology.orchestra.run.vm01.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:25.798 INFO:teuthology.orchestra.run.vm01.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-03-20T18:00:25.798 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:25.952 INFO:teuthology.orchestra.run.vm01.stdout: Installing : rbd-mirror-2:20.2.0-712.g70f8415b.el9.x86_64 131/140 2026-03-20T18:00:25.974 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: rbd-mirror-2:20.2.0-712.g70f8415b.el9.x86_64 131/140 2026-03-20T18:00:25.974 INFO:teuthology.orchestra.run.vm01.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-20T18:00:25.974 INFO:teuthology.orchestra.run.vm01.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-03-20T18:00:25.974 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-03-20T18:00:25.974 INFO:teuthology.orchestra.run.vm01.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-03-20T18:00:25.974 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:30.182 INFO:teuthology.orchestra.run.vm01.stdout: Installing : ceph-test-2:20.2.0-712.g70f8415b.el9.x86_64 132/140 2026-03-20T18:00:30.189 INFO:teuthology.orchestra.run.vm01.stdout: Installing : perl-Test-Harness-1:3.42-461.el9.noarch 133/140 2026-03-20T18:00:30.195 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libcephfs-devel-2:20.2.0-712.g70f8415b.el9.x86_6 134/140 2026-03-20T18:00:30.206 INFO:teuthology.orchestra.run.vm01.stdout: Installing : rbd-fuse-2:20.2.0-712.g70f8415b.el9.x86_64 135/140 2026-03-20T18:00:30.223 INFO:teuthology.orchestra.run.vm01.stdout: Installing : rbd-nbd-2:20.2.0-712.g70f8415b.el9.x86_64 136/140 2026-03-20T18:00:30.230 INFO:teuthology.orchestra.run.vm01.stdout: Installing : s3cmd-2.4.0-1.el9.noarch 137/140 2026-03-20T18:00:30.233 INFO:teuthology.orchestra.run.vm01.stdout: Installing : bzip2-1.0.8-11.el9.x86_64 138/140 2026-03-20T18:00:30.233 INFO:teuthology.orchestra.run.vm01.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 139/140 2026-03-20T18:00:30.248 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 139/140 2026-03-20T18:00:30.248 INFO:teuthology.orchestra.run.vm01.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 140/140 2026-03-20T18:00:31.688 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 140/140 2026-03-20T18:00:31.688 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-2:20.2.0-712.g70f8415b.el9.x86_64 1/140 2026-03-20T18:00:31.688 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-base-2:20.2.0-712.g70f8415b.el9.x86_64 2/140 2026-03-20T18:00:31.688 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-common-2:20.2.0-712.g70f8415b.el9.x86_64 3/140 2026-03-20T18:00:31.688 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-fuse-2:20.2.0-712.g70f8415b.el9.x86_64 4/140 2026-03-20T18:00:31.688 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-immutable-object-cache-2:20.2.0-712.g70f841 5/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-mds-2:20.2.0-712.g70f8415b.el9.x86_64 6/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-mgr-2:20.2.0-712.g70f8415b.el9.x86_64 7/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-mon-2:20.2.0-712.g70f8415b.el9.x86_64 8/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-osd-2:20.2.0-712.g70f8415b.el9.x86_64 9/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-radosgw-2:20.2.0-712.g70f8415b.el9.x86_64 10/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-selinux-2:20.2.0-712.g70f8415b.el9.x86_64 11/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-test-2:20.2.0-712.g70f8415b.el9.x86_64 12/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libcephfs-devel-2:20.2.0-712.g70f8415b.el9.x86_6 13/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libcephfs-proxy2-2:20.2.0-712.g70f8415b.el9.x86_ 14/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libcephfs2-2:20.2.0-712.g70f8415b.el9.x86_64 15/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libcephsqlite-2:20.2.0-712.g70f8415b.el9.x86_64 16/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : librados-devel-2:20.2.0-712.g70f8415b.el9.x86_64 17/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libradosstriper1-2:20.2.0-712.g70f8415b.el9.x86_ 18/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : librgw2-2:20.2.0-712.g70f8415b.el9.x86_64 19/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-ceph-argparse-2:20.2.0-712.g70f8415b.el9 20/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-ceph-common-2:20.2.0-712.g70f8415b.el9.x 21/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-cephfs-2:20.2.0-712.g70f8415b.el9.x86_64 22/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-rados-2:20.2.0-712.g70f8415b.el9.x86_64 23/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-rbd-2:20.2.0-712.g70f8415b.el9.x86_64 24/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-rgw-2:20.2.0-712.g70f8415b.el9.x86_64 25/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : rbd-fuse-2:20.2.0-712.g70f8415b.el9.x86_64 26/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : rbd-mirror-2:20.2.0-712.g70f8415b.el9.x86_64 27/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : rbd-nbd-2:20.2.0-712.g70f8415b.el9.x86_64 28/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-grafana-dashboards-2:20.2.0-712.g70f8415b.e 29/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-mgr-cephadm-2:20.2.0-712.g70f8415b.el9.noar 30/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-mgr-dashboard-2:20.2.0-712.g70f8415b.el9.no 31/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.2.0-712.g70f8 32/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-mgr-modules-core-2:20.2.0-712.g70f8415b.el9 33/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-mgr-rook-2:20.2.0-712.g70f8415b.el9.noarch 34/140 2026-03-20T18:00:31.689 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-prometheus-alerts-2:20.2.0-712.g70f8415b.el 35/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ceph-volume-2:20.2.0-712.g70f8415b.el9.noarch 36/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : cephadm-2:20.2.0-712.g70f8415b.el9.noarch 37/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : bzip2-1.0.8-11.el9.x86_64 38/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 39/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : fuse-2.9.9-17.el9.x86_64 40/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 41/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 42/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 43/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 44/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 45/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 46/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 47/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 48/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-ply-3.11-14.el9.noarch 49/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 50/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 51/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 52/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 53/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : unzip-6.0-59.el9.x86_64 54/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : zip-3.0-35.el9.x86_64 55/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 56/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 57/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 58/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 59/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 60/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 61/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 62/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 63/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 64/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 65/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 66/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : lua-5.4.4-4.el9.x86_64 67/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 68/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 69/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : perl-Benchmark-1.23-483.el9.noarch 70/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : perl-Test-Harness-1:3.42-461.el9.noarch 71/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 72/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 73/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 74/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 75/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 76/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 77/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 78/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 79/140 2026-03-20T18:00:31.692 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 80/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 81/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 82/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 83/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 84/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 86/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 87/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 88/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 89/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 90/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 91/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 92/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 93/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 94/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 95/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 96/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 97/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 98/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 99/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 100/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 102/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 103/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 104/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 105/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 106/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 107/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 108/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 109/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 110/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 111/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 112/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 113/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 114/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 115/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 116/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 117/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 118/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 119/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 120/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 121/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 122/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 123/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 124/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 125/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 126/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 127/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 128/140 2026-03-20T18:00:31.693 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 129/140 2026-03-20T18:00:31.694 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 130/140 2026-03-20T18:00:31.694 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/140 2026-03-20T18:00:31.694 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 132/140 2026-03-20T18:00:31.694 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/140 2026-03-20T18:00:31.694 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : re2-1:20211101-20.el9.x86_64 134/140 2026-03-20T18:00:31.694 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : s3cmd-2.4.0-1.el9.noarch 135/140 2026-03-20T18:00:31.694 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 136/140 2026-03-20T18:00:31.694 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : librados2-2:20.2.0-712.g70f8415b.el9.x86_64 137/140 2026-03-20T18:00:31.694 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 138/140 2026-03-20T18:00:31.694 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : librbd1-2:20.2.0-712.g70f8415b.el9.x86_64 139/140 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 140/140 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout:Upgraded: 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: librados2-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: librbd1-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout:Installed: 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: bzip2-1.0.8-11.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-base-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-common-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-fuse-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-grafana-dashboards-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-immutable-object-cache-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mds-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mgr-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mgr-cephadm-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mgr-dashboard-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mgr-diskprediction-local-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mgr-modules-core-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mgr-rook-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-mon-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-osd-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-prometheus-alerts-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-radosgw-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-selinux-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-test-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ceph-volume-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: cephadm-2:20.2.0-712.g70f8415b.el9.noarch 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: fuse-2.9.9-17.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: grpc-data-1.46.7-10.el9.noarch 2026-03-20T18:00:31.790 INFO:teuthology.orchestra.run.vm01.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libarrow-9.0.0-15.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libcephfs-devel-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libcephfs-proxy2-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libcephfs2-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libcephsqlite-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libconfig-1.7.2-9.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libnbd-1.20.3-4.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: liboath-2.6.12-1.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: librados-devel-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libradosstriper1-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: librgw2-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libunwind-1.6.2-1.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: libxslt-1.1.34-12.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: lua-5.4.4-4.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: luarocks-3.9.2-5.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: mailcap-2.1.49-5.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: openblas-0.3.29-1.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: pciutils-3.7.0-7.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: perl-Benchmark-1.23-483.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: perl-Test-Harness-1:3.42-461.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: protobuf-3.14.0-17.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-babel-2.9.1-2.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-ceph-argparse-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-ceph-common-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-cephfs-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-03-20T18:00:31.791 INFO:teuthology.orchestra.run.vm01.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-packaging-20.9-5.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-ply-3.11-14.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-portend-3.1.0-2.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-pycparser-2.20-6.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-rados-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-rbd-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-requests-2.25.1-10.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-rgw-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-routes-2.5.1-5.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-rsa-4.9-2.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-toml-0.10.2-6.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: qatlib-25.08.0-2.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: rbd-fuse-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: rbd-mirror-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: rbd-nbd-2:20.2.0-712.g70f8415b.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: re2-1:20211101-20.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: s3cmd-2.4.0-1.el9.noarch 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: socat-1.7.4.1-8.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: thrift-0.15.0-4.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: unzip-6.0-59.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: zip-3.0-35.el9.x86_64 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:31.792 INFO:teuthology.orchestra.run.vm01.stdout:Complete! 2026-03-20T18:00:31.880 DEBUG:teuthology.parallel:result is None 2026-03-20T18:00:31.880 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=70f8415b300f041766fa27faf7d5472699e32388 2026-03-20T18:00:32.479 DEBUG:teuthology.orchestra.run.vm01:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-03-20T18:00:32.497 INFO:teuthology.orchestra.run.vm01.stdout:20.2.0-712.g70f8415b.el9 2026-03-20T18:00:32.498 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-712.g70f8415b.el9 2026-03-20T18:00:32.498 INFO:teuthology.task.install:The correct ceph version 20.2.0-712.g70f8415b is installed. 2026-03-20T18:00:32.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=70f8415b300f041766fa27faf7d5472699e32388 2026-03-20T18:00:33.105 DEBUG:teuthology.orchestra.run.vm03:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-03-20T18:00:33.123 INFO:teuthology.orchestra.run.vm03.stdout:20.2.0-712.g70f8415b.el9 2026-03-20T18:00:33.123 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-712.g70f8415b.el9 2026-03-20T18:00:33.123 INFO:teuthology.task.install:The correct ceph version 20.2.0-712.g70f8415b is installed. 2026-03-20T18:00:33.124 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-03-20T18:00:33.124 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:33.124 DEBUG:teuthology.orchestra.run.vm01:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-03-20T18:00:33.148 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:33.149 DEBUG:teuthology.orchestra.run.vm03:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-03-20T18:00:33.188 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-03-20T18:00:33.188 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:33.188 DEBUG:teuthology.orchestra.run.vm01:> sudo dd of=/usr/bin/daemon-helper 2026-03-20T18:00:33.211 DEBUG:teuthology.orchestra.run.vm01:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-03-20T18:00:33.273 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:33.273 DEBUG:teuthology.orchestra.run.vm03:> sudo dd of=/usr/bin/daemon-helper 2026-03-20T18:00:33.296 DEBUG:teuthology.orchestra.run.vm03:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-03-20T18:00:33.357 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-03-20T18:00:33.357 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:33.357 DEBUG:teuthology.orchestra.run.vm01:> sudo dd of=/usr/bin/adjust-ulimits 2026-03-20T18:00:33.379 DEBUG:teuthology.orchestra.run.vm01:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-03-20T18:00:33.445 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:33.445 DEBUG:teuthology.orchestra.run.vm03:> sudo dd of=/usr/bin/adjust-ulimits 2026-03-20T18:00:33.468 DEBUG:teuthology.orchestra.run.vm03:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-03-20T18:00:33.531 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-03-20T18:00:33.531 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:33.531 DEBUG:teuthology.orchestra.run.vm01:> sudo dd of=/usr/bin/stdin-killer 2026-03-20T18:00:33.555 DEBUG:teuthology.orchestra.run.vm01:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-03-20T18:00:33.622 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:33.623 DEBUG:teuthology.orchestra.run.vm03:> sudo dd of=/usr/bin/stdin-killer 2026-03-20T18:00:33.646 DEBUG:teuthology.orchestra.run.vm03:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-03-20T18:00:33.711 INFO:teuthology.run_tasks:Running task ceph... 2026-03-20T18:00:33.750 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-03-20T18:00:33.750 DEBUG:teuthology.orchestra.run.vm01:> sudo chmod 777 /var/log/ceph 2026-03-20T18:00:33.752 DEBUG:teuthology.orchestra.run.vm03:> sudo chmod 777 /var/log/ceph 2026-03-20T18:00:33.778 INFO:tasks.ceph:Disabling ceph logrotate... 2026-03-20T18:00:33.778 DEBUG:teuthology.orchestra.run.vm01:> sudo rm -f -- /etc/logrotate.d/ceph 2026-03-20T18:00:33.819 DEBUG:teuthology.orchestra.run.vm03:> sudo rm -f -- /etc/logrotate.d/ceph 2026-03-20T18:00:33.845 INFO:tasks.ceph:Creating extra log directories... 2026-03-20T18:00:33.845 DEBUG:teuthology.orchestra.run.vm01:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-03-20T18:00:33.886 DEBUG:teuthology.orchestra.run.vm03:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-03-20T18:00:33.911 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-03-20T18:00:33.911 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw notification': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd mclock iops capacity threshold hdd': 49000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-03-20T18:00:33.911 INFO:tasks.ceph:ctx.config {'archive_path': '/archive/kyr-2026-03-20_17:25:16-rgw-tentacle-none-default-vps/2230', 'branch': 'tentacle', 'description': 'rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/amqp/{0-install centos_latest test_amqp} valgrind}', 'email': None, 'first_in_suite': False, 'flavor': 'default', 'job_id': '2230', 'last_in_suite': False, 'machine_type': 'vps', 'name': 'kyr-2026-03-20_17:25:16-rgw-tentacle-none-default-vps', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'tentacle'}, 'ansible.cephlab': {'branch': 'main', 'repo': 'https://github.com/kshtsk/ceph-cm-ansible.git', 'skip_tags': 'nagios,monitoring-scripts,hostname,pubkeys,zap,sudoers,kerberos,ntp-client,resolvconf,cpan,nfs', 'vars': {'logical_volumes': {'lv_1': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_2': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_3': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_4': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}}, 'timezone': 'UTC', 'volume_groups': {'vg_nvme': {'pvs': '/dev/vdb,/dev/vdc,/dev/vdd,/dev/vde'}}}}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw notification': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd mclock iops capacity threshold hdd': 49000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '70f8415b300f041766fa27faf7d5472699e32388'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'cephadm': {'cephadm_binary_url': 'https://download.ceph.com/rpm-20.2.0/el9/noarch/cephadm'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '70f8415b300f041766fa27faf7d5472699e32388'}, 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}}, 'rgw': {'client.0': {'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'client.1': {'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'frontend': 'beast', 'realm': 'MyRealm', 'storage classes': {'FROZEN': None, 'LUKEWARM': None}, 'zone': 'MyZone', 'zonegroup': 'MyZoneGroup'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'tt-tentacle', 'sha1': 'c04fae7bd18063e6f8f9e1cf40c23f810c1f6ca4'}}, 'owner': 'kyr', 'priority': 1000, 'repo': 'https://github.com/ceph/ceph.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 5336, 'sha1': '70f8415b300f041766fa27faf7d5472699e32388', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'tt-tentacle', 'suite_path': '/home/teuthos/src/github.com_kshtsk_ceph_c04fae7bd18063e6f8f9e1cf40c23f810c1f6ca4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/kshtsk/ceph.git', 'suite_sha1': 'c04fae7bd18063e6f8f9e1cf40c23f810c1f6ca4', 'targets': {'vm01.local': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPM2K9GoxhY4gNP+c5/VZH2o9RRCjh6T6aeNvsve1YRthc4ggz38D3QZ8AGNKxSajVe0BXWgGrDQaoDSTLWhEKk=', 'vm03.local': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCubnQzHt2zSKsan2cmD2y4ZJofSCUgPlfmIiloKSdwKBwzFDR7uruYPmhW3I5f5xx8/aHnYo4KtqyanMsoJvp4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'openssl_keys': None}, {'rgw': {'client.0': None}}, {'rabbitmq': {'client.0': None}}, {'notification-tests': {'client.0': {'extra_attr': ['amqp_test'], 'rgw_server': 'client.0'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'clyso-debian-13', 'teuthology_repo': 'https://github.com/clyso/teuthology', 'teuthology_sha1': '1c580df7a9c7c2aadc272da296344fd99f27c444', 'timestamp': '2026-03-20_17:25:16', 'tube': 'vps', 'user': 'kyr', 'verbose': False, 'worker_log': '/home/teuthos/.teuthology/dispatcher/dispatcher.vps.4188345'} 2026-03-20T18:00:33.911 DEBUG:teuthology.orchestra.run.vm01:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-03-20T18:00:33.948 DEBUG:teuthology.orchestra.run.vm03:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-03-20T18:00:33.962 DEBUG:teuthology.orchestra.run.vm01:> sudo install -d -m0777 -- /var/run/ceph 2026-03-20T18:00:34.003 DEBUG:teuthology.orchestra.run.vm03:> sudo install -d -m0777 -- /var/run/ceph 2026-03-20T18:00:34.026 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:34.026 DEBUG:teuthology.orchestra.run.vm01:> dd if=/scratch_devs of=/dev/stdout 2026-03-20T18:00:34.083 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-03-20T18:00:34.084 DEBUG:teuthology.orchestra.run.vm01:> stat /dev/vg_nvme/lv_1 2026-03-20T18:00:34.140 INFO:teuthology.orchestra.run.vm01.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-03-20T18:00:34.141 INFO:teuthology.orchestra.run.vm01.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-03-20T18:00:34.141 INFO:teuthology.orchestra.run.vm01.stdout:Device: 6h/6d Inode: 651 Links: 1 2026-03-20T18:00:34.141 INFO:teuthology.orchestra.run.vm01.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-03-20T18:00:34.141 INFO:teuthology.orchestra.run.vm01.stdout:Context: system_u:object_r:device_t:s0 2026-03-20T18:00:34.141 INFO:teuthology.orchestra.run.vm01.stdout:Access: 2026-03-20 18:00:30.507694164 +0000 2026-03-20T18:00:34.141 INFO:teuthology.orchestra.run.vm01.stdout:Modify: 2026-03-20 17:59:10.819615198 +0000 2026-03-20T18:00:34.141 INFO:teuthology.orchestra.run.vm01.stdout:Change: 2026-03-20 17:59:10.819615198 +0000 2026-03-20T18:00:34.141 INFO:teuthology.orchestra.run.vm01.stdout: Birth: 2026-03-20 17:59:10.819615198 +0000 2026-03-20T18:00:34.141 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-03-20T18:00:34.206 INFO:teuthology.orchestra.run.vm01.stderr:1+0 records in 2026-03-20T18:00:34.206 INFO:teuthology.orchestra.run.vm01.stderr:1+0 records out 2026-03-20T18:00:34.206 INFO:teuthology.orchestra.run.vm01.stderr:512 bytes copied, 0.000134111 s, 3.8 MB/s 2026-03-20T18:00:34.207 DEBUG:teuthology.orchestra.run.vm01:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-03-20T18:00:34.264 DEBUG:teuthology.orchestra.run.vm01:> stat /dev/vg_nvme/lv_2 2026-03-20T18:00:34.322 INFO:teuthology.orchestra.run.vm01.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-03-20T18:00:34.322 INFO:teuthology.orchestra.run.vm01.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-03-20T18:00:34.322 INFO:teuthology.orchestra.run.vm01.stdout:Device: 6h/6d Inode: 696 Links: 1 2026-03-20T18:00:34.322 INFO:teuthology.orchestra.run.vm01.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-03-20T18:00:34.322 INFO:teuthology.orchestra.run.vm01.stdout:Context: system_u:object_r:device_t:s0 2026-03-20T18:00:34.322 INFO:teuthology.orchestra.run.vm01.stdout:Access: 2026-03-20 18:00:30.507694164 +0000 2026-03-20T18:00:34.322 INFO:teuthology.orchestra.run.vm01.stdout:Modify: 2026-03-20 17:59:11.049613460 +0000 2026-03-20T18:00:34.322 INFO:teuthology.orchestra.run.vm01.stdout:Change: 2026-03-20 17:59:11.049613460 +0000 2026-03-20T18:00:34.322 INFO:teuthology.orchestra.run.vm01.stdout: Birth: 2026-03-20 17:59:11.049613460 +0000 2026-03-20T18:00:34.322 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-03-20T18:00:34.387 INFO:teuthology.orchestra.run.vm01.stderr:1+0 records in 2026-03-20T18:00:34.387 INFO:teuthology.orchestra.run.vm01.stderr:1+0 records out 2026-03-20T18:00:34.387 INFO:teuthology.orchestra.run.vm01.stderr:512 bytes copied, 0.000136887 s, 3.7 MB/s 2026-03-20T18:00:34.388 DEBUG:teuthology.orchestra.run.vm01:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-03-20T18:00:34.444 DEBUG:teuthology.orchestra.run.vm01:> stat /dev/vg_nvme/lv_3 2026-03-20T18:00:34.499 INFO:teuthology.orchestra.run.vm01.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-03-20T18:00:34.499 INFO:teuthology.orchestra.run.vm01.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-03-20T18:00:34.500 INFO:teuthology.orchestra.run.vm01.stdout:Device: 6h/6d Inode: 735 Links: 1 2026-03-20T18:00:34.500 INFO:teuthology.orchestra.run.vm01.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-03-20T18:00:34.500 INFO:teuthology.orchestra.run.vm01.stdout:Context: system_u:object_r:device_t:s0 2026-03-20T18:00:34.500 INFO:teuthology.orchestra.run.vm01.stdout:Access: 2026-03-20 18:00:30.508694165 +0000 2026-03-20T18:00:34.500 INFO:teuthology.orchestra.run.vm01.stdout:Modify: 2026-03-20 17:59:11.261611858 +0000 2026-03-20T18:00:34.500 INFO:teuthology.orchestra.run.vm01.stdout:Change: 2026-03-20 17:59:11.261611858 +0000 2026-03-20T18:00:34.500 INFO:teuthology.orchestra.run.vm01.stdout: Birth: 2026-03-20 17:59:11.261611858 +0000 2026-03-20T18:00:34.500 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-03-20T18:00:34.563 INFO:teuthology.orchestra.run.vm01.stderr:1+0 records in 2026-03-20T18:00:34.563 INFO:teuthology.orchestra.run.vm01.stderr:1+0 records out 2026-03-20T18:00:34.563 INFO:teuthology.orchestra.run.vm01.stderr:512 bytes copied, 0.000129843 s, 3.9 MB/s 2026-03-20T18:00:34.564 DEBUG:teuthology.orchestra.run.vm01:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-03-20T18:00:34.619 DEBUG:teuthology.orchestra.run.vm01:> stat /dev/vg_nvme/lv_4 2026-03-20T18:00:34.675 INFO:teuthology.orchestra.run.vm01.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-03-20T18:00:34.675 INFO:teuthology.orchestra.run.vm01.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-03-20T18:00:34.675 INFO:teuthology.orchestra.run.vm01.stdout:Device: 6h/6d Inode: 752 Links: 1 2026-03-20T18:00:34.675 INFO:teuthology.orchestra.run.vm01.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-03-20T18:00:34.675 INFO:teuthology.orchestra.run.vm01.stdout:Context: system_u:object_r:device_t:s0 2026-03-20T18:00:34.675 INFO:teuthology.orchestra.run.vm01.stdout:Access: 2026-03-20 18:00:30.508694165 +0000 2026-03-20T18:00:34.675 INFO:teuthology.orchestra.run.vm01.stdout:Modify: 2026-03-20 17:59:11.479610211 +0000 2026-03-20T18:00:34.675 INFO:teuthology.orchestra.run.vm01.stdout:Change: 2026-03-20 17:59:11.479610211 +0000 2026-03-20T18:00:34.675 INFO:teuthology.orchestra.run.vm01.stdout: Birth: 2026-03-20 17:59:11.479610211 +0000 2026-03-20T18:00:34.675 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-03-20T18:00:34.737 INFO:teuthology.orchestra.run.vm01.stderr:1+0 records in 2026-03-20T18:00:34.738 INFO:teuthology.orchestra.run.vm01.stderr:1+0 records out 2026-03-20T18:00:34.738 INFO:teuthology.orchestra.run.vm01.stderr:512 bytes copied, 0.000134792 s, 3.8 MB/s 2026-03-20T18:00:34.739 DEBUG:teuthology.orchestra.run.vm01:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-03-20T18:00:34.794 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-03-20T18:00:34.795 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:34.795 DEBUG:teuthology.orchestra.run.vm03:> dd if=/scratch_devs of=/dev/stdout 2026-03-20T18:00:34.809 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-03-20T18:00:34.809 DEBUG:teuthology.orchestra.run.vm03:> stat /dev/vg_nvme/lv_1 2026-03-20T18:00:34.867 INFO:teuthology.orchestra.run.vm03.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-03-20T18:00:34.867 INFO:teuthology.orchestra.run.vm03.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-03-20T18:00:34.867 INFO:teuthology.orchestra.run.vm03.stdout:Device: 6h/6d Inode: 645 Links: 1 2026-03-20T18:00:34.867 INFO:teuthology.orchestra.run.vm03.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-03-20T18:00:34.867 INFO:teuthology.orchestra.run.vm03.stdout:Context: system_u:object_r:device_t:s0 2026-03-20T18:00:34.867 INFO:teuthology.orchestra.run.vm03.stdout:Access: 2026-03-20 18:00:21.329564671 +0000 2026-03-20T18:00:34.867 INFO:teuthology.orchestra.run.vm03.stdout:Modify: 2026-03-20 17:59:10.193781038 +0000 2026-03-20T18:00:34.867 INFO:teuthology.orchestra.run.vm03.stdout:Change: 2026-03-20 17:59:10.193781038 +0000 2026-03-20T18:00:34.867 INFO:teuthology.orchestra.run.vm03.stdout: Birth: 2026-03-20 17:59:10.193781038 +0000 2026-03-20T18:00:34.867 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-03-20T18:00:34.930 INFO:teuthology.orchestra.run.vm03.stderr:1+0 records in 2026-03-20T18:00:34.930 INFO:teuthology.orchestra.run.vm03.stderr:1+0 records out 2026-03-20T18:00:34.930 INFO:teuthology.orchestra.run.vm03.stderr:512 bytes copied, 0.000151443 s, 3.4 MB/s 2026-03-20T18:00:34.931 DEBUG:teuthology.orchestra.run.vm03:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-03-20T18:00:34.989 DEBUG:teuthology.orchestra.run.vm03:> stat /dev/vg_nvme/lv_2 2026-03-20T18:00:35.048 INFO:teuthology.orchestra.run.vm03.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-03-20T18:00:35.048 INFO:teuthology.orchestra.run.vm03.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-03-20T18:00:35.048 INFO:teuthology.orchestra.run.vm03.stdout:Device: 6h/6d Inode: 688 Links: 1 2026-03-20T18:00:35.049 INFO:teuthology.orchestra.run.vm03.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-03-20T18:00:35.049 INFO:teuthology.orchestra.run.vm03.stdout:Context: system_u:object_r:device_t:s0 2026-03-20T18:00:35.049 INFO:teuthology.orchestra.run.vm03.stdout:Access: 2026-03-20 18:00:21.329564671 +0000 2026-03-20T18:00:35.049 INFO:teuthology.orchestra.run.vm03.stdout:Modify: 2026-03-20 17:59:10.430788828 +0000 2026-03-20T18:00:35.049 INFO:teuthology.orchestra.run.vm03.stdout:Change: 2026-03-20 17:59:10.430788828 +0000 2026-03-20T18:00:35.049 INFO:teuthology.orchestra.run.vm03.stdout: Birth: 2026-03-20 17:59:10.430788828 +0000 2026-03-20T18:00:35.049 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-03-20T18:00:35.112 INFO:teuthology.orchestra.run.vm03.stderr:1+0 records in 2026-03-20T18:00:35.112 INFO:teuthology.orchestra.run.vm03.stderr:1+0 records out 2026-03-20T18:00:35.112 INFO:teuthology.orchestra.run.vm03.stderr:512 bytes copied, 0.000118742 s, 4.3 MB/s 2026-03-20T18:00:35.113 DEBUG:teuthology.orchestra.run.vm03:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-03-20T18:00:35.169 DEBUG:teuthology.orchestra.run.vm03:> stat /dev/vg_nvme/lv_3 2026-03-20T18:00:35.226 INFO:teuthology.orchestra.run.vm03.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-03-20T18:00:35.226 INFO:teuthology.orchestra.run.vm03.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-03-20T18:00:35.226 INFO:teuthology.orchestra.run.vm03.stdout:Device: 6h/6d Inode: 737 Links: 1 2026-03-20T18:00:35.226 INFO:teuthology.orchestra.run.vm03.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-03-20T18:00:35.226 INFO:teuthology.orchestra.run.vm03.stdout:Context: system_u:object_r:device_t:s0 2026-03-20T18:00:35.226 INFO:teuthology.orchestra.run.vm03.stdout:Access: 2026-03-20 18:00:21.329564671 +0000 2026-03-20T18:00:35.226 INFO:teuthology.orchestra.run.vm03.stdout:Modify: 2026-03-20 17:59:10.650796060 +0000 2026-03-20T18:00:35.226 INFO:teuthology.orchestra.run.vm03.stdout:Change: 2026-03-20 17:59:10.650796060 +0000 2026-03-20T18:00:35.226 INFO:teuthology.orchestra.run.vm03.stdout: Birth: 2026-03-20 17:59:10.650796060 +0000 2026-03-20T18:00:35.226 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-03-20T18:00:35.288 INFO:teuthology.orchestra.run.vm03.stderr:1+0 records in 2026-03-20T18:00:35.288 INFO:teuthology.orchestra.run.vm03.stderr:1+0 records out 2026-03-20T18:00:35.288 INFO:teuthology.orchestra.run.vm03.stderr:512 bytes copied, 0.000110918 s, 4.6 MB/s 2026-03-20T18:00:35.289 DEBUG:teuthology.orchestra.run.vm03:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-03-20T18:00:35.348 DEBUG:teuthology.orchestra.run.vm03:> stat /dev/vg_nvme/lv_4 2026-03-20T18:00:35.405 INFO:teuthology.orchestra.run.vm03.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-03-20T18:00:35.405 INFO:teuthology.orchestra.run.vm03.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-03-20T18:00:35.405 INFO:teuthology.orchestra.run.vm03.stdout:Device: 6h/6d Inode: 768 Links: 1 2026-03-20T18:00:35.405 INFO:teuthology.orchestra.run.vm03.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-03-20T18:00:35.405 INFO:teuthology.orchestra.run.vm03.stdout:Context: system_u:object_r:device_t:s0 2026-03-20T18:00:35.405 INFO:teuthology.orchestra.run.vm03.stdout:Access: 2026-03-20 18:00:21.330564673 +0000 2026-03-20T18:00:35.405 INFO:teuthology.orchestra.run.vm03.stdout:Modify: 2026-03-20 17:59:10.889803916 +0000 2026-03-20T18:00:35.405 INFO:teuthology.orchestra.run.vm03.stdout:Change: 2026-03-20 17:59:10.889803916 +0000 2026-03-20T18:00:35.405 INFO:teuthology.orchestra.run.vm03.stdout: Birth: 2026-03-20 17:59:10.889803916 +0000 2026-03-20T18:00:35.405 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-03-20T18:00:35.471 INFO:teuthology.orchestra.run.vm03.stderr:1+0 records in 2026-03-20T18:00:35.471 INFO:teuthology.orchestra.run.vm03.stderr:1+0 records out 2026-03-20T18:00:35.471 INFO:teuthology.orchestra.run.vm03.stderr:512 bytes copied, 0.000127959 s, 4.0 MB/s 2026-03-20T18:00:35.472 DEBUG:teuthology.orchestra.run.vm03:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-03-20T18:00:35.528 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2026-03-20T18:00:35.529 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@vm01.local'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@vm03.local'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2026-03-20T18:00:35.529 INFO:tasks.ceph:Generating config... 2026-03-20T18:00:35.529 INFO:tasks.ceph:[client] debug rgw = 20 2026-03-20T18:00:35.529 INFO:tasks.ceph:[client] debug rgw lifecycle = 20 2026-03-20T18:00:35.529 INFO:tasks.ceph:[client] debug rgw notification = 20 2026-03-20T18:00:35.529 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2026-03-20T18:00:35.529 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2026-03-20T18:00:35.529 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2026-03-20T18:00:35.529 INFO:tasks.ceph:[client] setgroup = ceph 2026-03-20T18:00:35.529 INFO:tasks.ceph:[client] setuser = ceph 2026-03-20T18:00:35.529 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 10 2026-03-20T18:00:35.529 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 10 2026-03-20T18:00:35.529 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-03-20T18:00:35.529 INFO:tasks.ceph:[mgr] debug ms = 1 2026-03-20T18:00:35.529 INFO:tasks.ceph:[mon] debug mon = 20 2026-03-20T18:00:35.529 INFO:tasks.ceph:[mon] debug ms = 1 2026-03-20T18:00:35.529 INFO:tasks.ceph:[mon] debug paxos = 20 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] bdev async discard = True 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] bdev enable discard = True 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] debug ms = 1 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] debug osd = 20 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-03-20T18:00:35.529 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-03-20T18:00:35.530 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-03-20T18:00:35.530 INFO:tasks.ceph:[osd] osd mclock iops capacity threshold hdd = 49000 2026-03-20T18:00:35.530 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-03-20T18:00:35.530 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-03-20T18:00:35.530 INFO:tasks.ceph:Setting up mon.a... 2026-03-20T18:00:35.530 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-03-20T18:00:35.566 INFO:teuthology.orchestra.run.vm01.stdout:creating /etc/ceph/ceph.keyring 2026-03-20T18:00:35.569 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-03-20T18:00:35.652 DEBUG:teuthology.orchestra.run.vm01:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-03-20T18:00:35.719 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '192.168.123.101'), ('mon.c', '[v2:192.168.123.101:3301,v1:192.168.123.101:6790]'), ('mon.b', '192.168.123.103')] 2026-03-20T18:00:35.719 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '192.168.123.101,[v2:192.168.123.101:3301,v1:192.168.123.101:6790],192.168.123.103', 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd mclock iops capacity threshold hdd': 49000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw notification': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2026-03-20T18:00:35.720 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:35.720 DEBUG:teuthology.orchestra.run.vm01:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-03-20T18:00:35.776 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 192.168.123.101 --addv c '[v2:192.168.123.101:3301,v1:192.168.123.101:6790]' --add b 192.168.123.103 --print /home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:monmaptool: generated fsid e5ff0a4c-c523-496b-bc3f-e531ec13e9f0 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:setting min_mon_release = tentacle 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:epoch 0 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:fsid e5ff0a4c-c523-496b-bc3f-e531ec13e9f0 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:last_changed 2026-03-20T18:00:35.858197+0000 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:created 2026-03-20T18:00:35.858197+0000 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:min_mon_release 20 (tentacle) 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:election_strategy: 1 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:0: [v2:192.168.123.101:3300/0,v1:192.168.123.101:6789/0] mon.a 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:1: [v2:192.168.123.103:3300/0,v1:192.168.123.103:6789/0] mon.b 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:2: [v2:192.168.123.101:3301/0,v1:192.168.123.101:6790/0] mon.c 2026-03-20T18:00:35.855 INFO:teuthology.orchestra.run.vm01.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2026-03-20T18:00:35.857 DEBUG:teuthology.orchestra.run.vm01:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-03-20T18:00:35.916 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e5ff0a4c-c523-496b-bc3f-e531ec13e9f0... 2026-03-20T18:00:35.917 DEBUG:teuthology.orchestra.run.vm01:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-03-20T18:00:35.959 DEBUG:teuthology.orchestra.run.vm03:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout:[global] 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: chdir = "" 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: auth supported = cephx 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: filestore xattr use omap = true 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: mon clock drift allowed = 1.000 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: osd crush chooseleaf type = 0 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: auth debug = true 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: ms die on old message = true 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: ms die on bug = true 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: mon max pg per osd = 10000 # >= luminous 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: mon pg warn max object skew = 0 2026-03-20T18:00:36.002 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: # disable pg_autoscaler by default for new pools 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd_pool_default_pg_autoscale_mode = off 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd pool default size = 2 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon osd allow primary affinity = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon osd allow pg remap = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon warn on legacy crush tunables = false 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon warn on crush straw calc version zero = false 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon warn on no sortbitwise = false 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon warn on osd down out interval zero = false 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon warn on too few osds = false 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon_warn_on_pool_no_redundancy = false 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon_allow_pool_size_one = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd default data pool replay window = 5 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon allow pool delete = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon cluster log file level = debug 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: debug asserts on shutdown = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon health detail to clog = false 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: mon host = "192.168.123.101,[v2:192.168.123.101:3301,v1:192.168.123.101:6790],192.168.123.103" 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd_max_pg_log_entries = 10 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd_min_pg_log_entries = 10 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: fsid = e5ff0a4c-c523-496b-bc3f-e531ec13e9f0 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout:[osd] 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd journal size = 100 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd scrub load threshold = 5.0 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd scrub max interval = 600 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd mclock profile = high_recovery_ops 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd mclock skip benchmark = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd recover clone overlap = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd recovery max chunk = 1048576 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd debug shutdown = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd debug op order = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd debug verify stray on activate = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd debug trim objects = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd open classes on start = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd debug pg log writeout = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd deep scrub update digest min age = 30 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd map max advance = 10 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: journal zero on create = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: filestore ondisk finisher threads = 3 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: filestore apply finisher threads = 3 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: bdev debug aio = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: osd debug misdirected ops = true 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: bdev async discard = True 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: bdev enable discard = True 2026-03-20T18:00:36.003 INFO:teuthology.orchestra.run.vm01.stdout: bluestore allocator = bitmap 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: bluestore block size = 96636764160 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: bluestore fsck on mount = True 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug bluefs = 1/20 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug bluestore = 1/20 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug ms = 1 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug osd = 20 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug rocksdb = 4/10 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon osd backfillfull_ratio = 0.85 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon osd full ratio = 0.9 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon osd nearfull ratio = 0.8 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: osd failsafe full ratio = 0.95 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: osd mclock iops capacity threshold hdd = 49000 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: osd objectstore = bluestore 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: osd shutdown pgref assert = True 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout:[mgr] 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug ms = 1 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug mgr = 20 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug mon = 20 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug auth = 20 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon reweight min pgs per osd = 4 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon reweight min bytes per osd = 10 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mgr/telemetry/nag = false 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout:[mon] 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug ms = 1 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug mon = 20 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug paxos = 20 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: debug auth = 20 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon data avail warn = 5 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon mgr mkfs grace = 240 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon reweight min pgs per osd = 4 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon osd reporter subtree level = osd 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon osd prime pg temp = true 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon reweight min bytes per osd = 10 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: auth mon ticket ttl = 660 # 11m 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: auth service ticket ttl = 240 # 4m 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: # don't complain about insecure global_id in the test suite 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: # 1m isn't quite enough 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon_down_mkfs_grace = 2m 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: mon_warn_on_filestore_osds = false 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout:[client] 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: rgw cache enabled = true 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: rgw enable ops log = true 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: rgw enable usage log = true 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-03-20T18:00:36.004 INFO:teuthology.orchestra.run.vm01.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-03-20T18:00:36.005 INFO:teuthology.orchestra.run.vm01.stdout: debug rgw = 20 2026-03-20T18:00:36.005 INFO:teuthology.orchestra.run.vm01.stdout: debug rgw lifecycle = 20 2026-03-20T18:00:36.005 INFO:teuthology.orchestra.run.vm01.stdout: debug rgw notification = 20 2026-03-20T18:00:36.005 INFO:teuthology.orchestra.run.vm01.stdout: rgw crypt require ssl = False 2026-03-20T18:00:36.005 INFO:teuthology.orchestra.run.vm01.stdout: rgw crypt s3 kms backend = testing 2026-03-20T18:00:36.005 INFO:teuthology.orchestra.run.vm01.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2026-03-20T18:00:36.005 INFO:teuthology.orchestra.run.vm01.stdout: setgroup = ceph 2026-03-20T18:00:36.005 INFO:teuthology.orchestra.run.vm01.stdout: setuser = ceph 2026-03-20T18:00:36.005 INFO:teuthology.orchestra.run.vm01.stdout:[mon.a] 2026-03-20T18:00:36.005 INFO:teuthology.orchestra.run.vm01.stdout:[mon.c] 2026-03-20T18:00:36.005 INFO:teuthology.orchestra.run.vm01.stdout:[mon.b] 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout:[global] 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: chdir = "" 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: auth supported = cephx 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: filestore xattr use omap = true 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: mon clock drift allowed = 1.000 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: osd crush chooseleaf type = 0 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: auth debug = true 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: ms die on old message = true 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: ms die on bug = true 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: mon max pg per osd = 10000 # >= luminous 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: mon pg warn max object skew = 0 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: # disable pg_autoscaler by default for new pools 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: osd_pool_default_pg_autoscale_mode = off 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: osd pool default size = 2 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: mon osd allow primary affinity = true 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: mon osd allow pg remap = true 2026-03-20T18:00:36.007 INFO:teuthology.orchestra.run.vm03.stdout: mon warn on legacy crush tunables = false 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: mon warn on crush straw calc version zero = false 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: mon warn on no sortbitwise = false 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: mon warn on osd down out interval zero = false 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: mon warn on too few osds = false 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: mon_warn_on_pool_no_redundancy = false 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: mon_allow_pool_size_one = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd default data pool replay window = 5 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: mon allow pool delete = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: mon cluster log file level = debug 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: debug asserts on shutdown = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: mon health detail to clog = false 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: mon host = "192.168.123.101,[v2:192.168.123.101:3301,v1:192.168.123.101:6790],192.168.123.103" 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd_max_pg_log_entries = 10 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd_min_pg_log_entries = 10 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: fsid = e5ff0a4c-c523-496b-bc3f-e531ec13e9f0 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout:[osd] 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd journal size = 100 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd scrub load threshold = 5.0 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd scrub max interval = 600 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd mclock profile = high_recovery_ops 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd mclock skip benchmark = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd recover clone overlap = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd recovery max chunk = 1048576 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd debug shutdown = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd debug op order = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd debug verify stray on activate = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd debug trim objects = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd open classes on start = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd debug pg log writeout = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd deep scrub update digest min age = 30 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd map max advance = 10 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: journal zero on create = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: filestore ondisk finisher threads = 3 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: filestore apply finisher threads = 3 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: bdev debug aio = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: osd debug misdirected ops = true 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: bdev async discard = True 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: bdev enable discard = True 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: bluestore allocator = bitmap 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: bluestore block size = 96636764160 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: bluestore fsck on mount = True 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: debug bluefs = 1/20 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: debug bluestore = 1/20 2026-03-20T18:00:36.008 INFO:teuthology.orchestra.run.vm03.stdout: debug ms = 1 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug osd = 20 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug rocksdb = 4/10 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon osd backfillfull_ratio = 0.85 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon osd full ratio = 0.9 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon osd nearfull ratio = 0.8 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: osd failsafe full ratio = 0.95 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: osd mclock iops capacity threshold hdd = 49000 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: osd objectstore = bluestore 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: osd shutdown pgref assert = True 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout:[mgr] 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug ms = 1 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug mgr = 20 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug mon = 20 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug auth = 20 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon reweight min pgs per osd = 4 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon reweight min bytes per osd = 10 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mgr/telemetry/nag = false 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout:[mon] 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug ms = 1 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug mon = 20 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug paxos = 20 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug auth = 20 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon data avail warn = 5 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon mgr mkfs grace = 240 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon reweight min pgs per osd = 4 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon osd reporter subtree level = osd 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon osd prime pg temp = true 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon reweight min bytes per osd = 10 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: auth mon ticket ttl = 660 # 11m 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: auth service ticket ttl = 240 # 4m 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: # don't complain about insecure global_id in the test suite 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: # 1m isn't quite enough 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon_down_mkfs_grace = 2m 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: mon_warn_on_filestore_osds = false 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout:[client] 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: rgw cache enabled = true 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: rgw enable ops log = true 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: rgw enable usage log = true 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug rgw = 20 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug rgw lifecycle = 20 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: debug rgw notification = 20 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: rgw crypt require ssl = False 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: rgw crypt s3 kms backend = testing 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: setgroup = ceph 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout: setuser = ceph 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout:[mon.a] 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout:[mon.c] 2026-03-20T18:00:36.009 INFO:teuthology.orchestra.run.vm03.stdout:[mon.b] 2026-03-20T18:00:36.020 INFO:tasks.ceph:Creating admin key on mon.a... 2026-03-20T18:00:36.020 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-03-20T18:00:36.099 INFO:tasks.ceph:Copying monmap to all nodes... 2026-03-20T18:00:36.099 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:36.099 DEBUG:teuthology.orchestra.run.vm01:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-03-20T18:00:36.113 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:36.113 DEBUG:teuthology.orchestra.run.vm01:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-03-20T18:00:36.168 INFO:tasks.ceph:Sending monmap to node ubuntu@vm01.local 2026-03-20T18:00:36.168 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:36.169 DEBUG:teuthology.orchestra.run.vm01:> sudo dd of=/etc/ceph/ceph.keyring 2026-03-20T18:00:36.169 DEBUG:teuthology.orchestra.run.vm01:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-03-20T18:00:36.244 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:36.244 DEBUG:teuthology.orchestra.run.vm01:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:36.300 INFO:tasks.ceph:Sending monmap to node ubuntu@vm03.local 2026-03-20T18:00:36.300 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:36.300 DEBUG:teuthology.orchestra.run.vm03:> sudo dd of=/etc/ceph/ceph.keyring 2026-03-20T18:00:36.300 DEBUG:teuthology.orchestra.run.vm03:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-03-20T18:00:36.336 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:36.336 DEBUG:teuthology.orchestra.run.vm03:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:36.391 INFO:tasks.ceph:Setting up mon nodes... 2026-03-20T18:00:36.391 INFO:tasks.ceph:Setting up mgr nodes... 2026-03-20T18:00:36.391 DEBUG:teuthology.orchestra.run.vm01:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2026-03-20T18:00:36.438 INFO:teuthology.orchestra.run.vm01.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2026-03-20T18:00:36.440 DEBUG:teuthology.orchestra.run.vm03:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-03-20T18:00:36.492 INFO:teuthology.orchestra.run.vm03.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-03-20T18:00:36.495 INFO:tasks.ceph:Setting up mds nodes... 2026-03-20T18:00:36.495 INFO:tasks.ceph_client:Setting up client nodes... 2026-03-20T18:00:36.495 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-03-20T18:00:36.535 INFO:teuthology.orchestra.run.vm01.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-03-20T18:00:36.548 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2026-03-20T18:00:36.585 INFO:teuthology.orchestra.run.vm03.stdout:creating /etc/ceph/ceph.client.1.keyring 2026-03-20T18:00:36.598 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-03-20T18:00:36.599 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@vm01.local'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@vm03.local'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2026-03-20T18:00:36.599 DEBUG:teuthology.orchestra.run.vm01:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-03-20T18:00:36.623 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-03-20T18:00:36.623 INFO:tasks.ceph:role: osd.0 2026-03-20T18:00:36.623 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@vm01.local 2026-03-20T18:00:36.623 DEBUG:teuthology.orchestra.run.vm01:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-03-20T18:00:36.691 INFO:teuthology.orchestra.run.vm01.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=1310464 blks 2026-03-20T18:00:36.691 INFO:teuthology.orchestra.run.vm01.stdout: = sectsz=512 attr=2, projid32bit=1 2026-03-20T18:00:36.691 INFO:teuthology.orchestra.run.vm01.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-03-20T18:00:36.691 INFO:teuthology.orchestra.run.vm01.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-03-20T18:00:36.691 INFO:teuthology.orchestra.run.vm01.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-03-20T18:00:36.691 INFO:teuthology.orchestra.run.vm01.stdout: = sunit=0 swidth=0 blks 2026-03-20T18:00:36.691 INFO:teuthology.orchestra.run.vm01.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-03-20T18:00:36.691 INFO:teuthology.orchestra.run.vm01.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-03-20T18:00:36.691 INFO:teuthology.orchestra.run.vm01.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-03-20T18:00:36.691 INFO:teuthology.orchestra.run.vm01.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-03-20T18:00:36.696 INFO:teuthology.orchestra.run.vm01.stdout:Discarding blocks...Done. 2026-03-20T18:00:36.700 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@vm01.local -o noatime 2026-03-20T18:00:36.700 DEBUG:teuthology.orchestra.run.vm01:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-03-20T18:00:36.771 DEBUG:teuthology.orchestra.run.vm01:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-03-20T18:00:36.839 DEBUG:teuthology.orchestra.run.vm01:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-03-20T18:00:36.909 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-03-20T18:00:36.909 INFO:tasks.ceph:role: osd.1 2026-03-20T18:00:36.909 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@vm01.local 2026-03-20T18:00:36.909 DEBUG:teuthology.orchestra.run.vm01:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-03-20T18:00:36.976 INFO:teuthology.orchestra.run.vm01.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=1310464 blks 2026-03-20T18:00:36.976 INFO:teuthology.orchestra.run.vm01.stdout: = sectsz=512 attr=2, projid32bit=1 2026-03-20T18:00:36.976 INFO:teuthology.orchestra.run.vm01.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-03-20T18:00:36.976 INFO:teuthology.orchestra.run.vm01.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-03-20T18:00:36.976 INFO:teuthology.orchestra.run.vm01.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-03-20T18:00:36.976 INFO:teuthology.orchestra.run.vm01.stdout: = sunit=0 swidth=0 blks 2026-03-20T18:00:36.976 INFO:teuthology.orchestra.run.vm01.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-03-20T18:00:36.976 INFO:teuthology.orchestra.run.vm01.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-03-20T18:00:36.976 INFO:teuthology.orchestra.run.vm01.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-03-20T18:00:36.976 INFO:teuthology.orchestra.run.vm01.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-03-20T18:00:36.981 INFO:teuthology.orchestra.run.vm01.stdout:Discarding blocks...Done. 2026-03-20T18:00:36.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@vm01.local -o noatime 2026-03-20T18:00:36.983 DEBUG:teuthology.orchestra.run.vm01:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-03-20T18:00:37.054 DEBUG:teuthology.orchestra.run.vm01:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-03-20T18:00:37.121 DEBUG:teuthology.orchestra.run.vm01:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-03-20T18:00:37.187 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-03-20T18:00:37.187 INFO:tasks.ceph:role: osd.2 2026-03-20T18:00:37.187 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@vm01.local 2026-03-20T18:00:37.187 DEBUG:teuthology.orchestra.run.vm01:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-03-20T18:00:37.252 INFO:teuthology.orchestra.run.vm01.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=1310464 blks 2026-03-20T18:00:37.252 INFO:teuthology.orchestra.run.vm01.stdout: = sectsz=512 attr=2, projid32bit=1 2026-03-20T18:00:37.252 INFO:teuthology.orchestra.run.vm01.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-03-20T18:00:37.252 INFO:teuthology.orchestra.run.vm01.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-03-20T18:00:37.252 INFO:teuthology.orchestra.run.vm01.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-03-20T18:00:37.252 INFO:teuthology.orchestra.run.vm01.stdout: = sunit=0 swidth=0 blks 2026-03-20T18:00:37.252 INFO:teuthology.orchestra.run.vm01.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-03-20T18:00:37.252 INFO:teuthology.orchestra.run.vm01.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-03-20T18:00:37.252 INFO:teuthology.orchestra.run.vm01.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-03-20T18:00:37.252 INFO:teuthology.orchestra.run.vm01.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-03-20T18:00:37.257 INFO:teuthology.orchestra.run.vm01.stdout:Discarding blocks...Done. 2026-03-20T18:00:37.258 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@vm01.local -o noatime 2026-03-20T18:00:37.258 DEBUG:teuthology.orchestra.run.vm01:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-03-20T18:00:37.330 DEBUG:teuthology.orchestra.run.vm01:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-03-20T18:00:37.400 DEBUG:teuthology.orchestra.run.vm01:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-03-20T18:00:37.466 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-03-20T18:00:37.466 INFO:tasks.ceph:role: osd.3 2026-03-20T18:00:37.466 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@vm01.local 2026-03-20T18:00:37.466 DEBUG:teuthology.orchestra.run.vm01:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-03-20T18:00:37.530 INFO:teuthology.orchestra.run.vm01.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=1310464 blks 2026-03-20T18:00:37.530 INFO:teuthology.orchestra.run.vm01.stdout: = sectsz=512 attr=2, projid32bit=1 2026-03-20T18:00:37.531 INFO:teuthology.orchestra.run.vm01.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-03-20T18:00:37.531 INFO:teuthology.orchestra.run.vm01.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-03-20T18:00:37.531 INFO:teuthology.orchestra.run.vm01.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-03-20T18:00:37.531 INFO:teuthology.orchestra.run.vm01.stdout: = sunit=0 swidth=0 blks 2026-03-20T18:00:37.531 INFO:teuthology.orchestra.run.vm01.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-03-20T18:00:37.531 INFO:teuthology.orchestra.run.vm01.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-03-20T18:00:37.531 INFO:teuthology.orchestra.run.vm01.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-03-20T18:00:37.531 INFO:teuthology.orchestra.run.vm01.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-03-20T18:00:37.535 INFO:teuthology.orchestra.run.vm01.stdout:Discarding blocks...Done. 2026-03-20T18:00:37.537 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@vm01.local -o noatime 2026-03-20T18:00:37.537 DEBUG:teuthology.orchestra.run.vm01:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2026-03-20T18:00:37.608 DEBUG:teuthology.orchestra.run.vm01:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-03-20T18:00:37.674 DEBUG:teuthology.orchestra.run.vm01:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:37.757 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:37.758+0000 7f75046f1900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-03-20T18:00:37.758 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:37.759+0000 7f75046f1900 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-03-20T18:00:37.758 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:37.759+0000 7f75046f1900 -1 bdev(0x560b7fd27800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-03-20T18:00:37.758 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:37.759+0000 7f75046f1900 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-03-20T18:00:38.445 DEBUG:teuthology.orchestra.run.vm01:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-03-20T18:00:38.515 DEBUG:teuthology.orchestra.run.vm01:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:38.597 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:38.598+0000 7f7325ba0900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-03-20T18:00:38.598 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:38.598+0000 7f7325ba0900 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-03-20T18:00:38.598 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:38.598+0000 7f7325ba0900 -1 bdev(0x5556b5627800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-03-20T18:00:38.598 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:38.598+0000 7f7325ba0900 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-03-20T18:00:39.307 DEBUG:teuthology.orchestra.run.vm01:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-03-20T18:00:39.375 DEBUG:teuthology.orchestra.run.vm01:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:39.459 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:39.460+0000 7fc5b5e0f900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-03-20T18:00:39.460 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:39.461+0000 7fc5b5e0f900 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-03-20T18:00:39.460 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:39.461+0000 7fc5b5e0f900 -1 bdev(0x5623b61c9800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-03-20T18:00:39.460 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:39.461+0000 7fc5b5e0f900 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-03-20T18:00:40.138 DEBUG:teuthology.orchestra.run.vm01:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-03-20T18:00:40.165 DEBUG:teuthology.orchestra.run.vm01:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:40.243 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:40.244+0000 7faed8364900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2026-03-20T18:00:40.243 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:40.244+0000 7faed8364900 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2026-03-20T18:00:40.243 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:40.244+0000 7faed8364900 -1 bdev(0x55dde6caf800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2026-03-20T18:00:40.243 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:00:40.244+0000 7faed8364900 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2026-03-20T18:00:40.907 DEBUG:teuthology.orchestra.run.vm01:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-03-20T18:00:40.973 DEBUG:teuthology.orchestra.run.vm03:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2026-03-20T18:00:40.998 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2026-03-20T18:00:40.998 INFO:tasks.ceph:role: osd.4 2026-03-20T18:00:40.998 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@vm03.local 2026-03-20T18:00:40.998 DEBUG:teuthology.orchestra.run.vm03:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-03-20T18:00:41.065 INFO:teuthology.orchestra.run.vm03.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=1310464 blks 2026-03-20T18:00:41.065 INFO:teuthology.orchestra.run.vm03.stdout: = sectsz=512 attr=2, projid32bit=1 2026-03-20T18:00:41.065 INFO:teuthology.orchestra.run.vm03.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-03-20T18:00:41.065 INFO:teuthology.orchestra.run.vm03.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-03-20T18:00:41.065 INFO:teuthology.orchestra.run.vm03.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-03-20T18:00:41.065 INFO:teuthology.orchestra.run.vm03.stdout: = sunit=0 swidth=0 blks 2026-03-20T18:00:41.065 INFO:teuthology.orchestra.run.vm03.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-03-20T18:00:41.065 INFO:teuthology.orchestra.run.vm03.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-03-20T18:00:41.065 INFO:teuthology.orchestra.run.vm03.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-03-20T18:00:41.065 INFO:teuthology.orchestra.run.vm03.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-03-20T18:00:41.070 INFO:teuthology.orchestra.run.vm03.stdout:Discarding blocks...Done. 2026-03-20T18:00:41.072 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@vm03.local -o noatime 2026-03-20T18:00:41.072 DEBUG:teuthology.orchestra.run.vm03:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2026-03-20T18:00:41.145 DEBUG:teuthology.orchestra.run.vm03:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2026-03-20T18:00:41.216 DEBUG:teuthology.orchestra.run.vm03:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2026-03-20T18:00:41.283 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2026-03-20T18:00:41.283 INFO:tasks.ceph:role: osd.5 2026-03-20T18:00:41.283 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@vm03.local 2026-03-20T18:00:41.283 DEBUG:teuthology.orchestra.run.vm03:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-03-20T18:00:41.350 INFO:teuthology.orchestra.run.vm03.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=1310464 blks 2026-03-20T18:00:41.351 INFO:teuthology.orchestra.run.vm03.stdout: = sectsz=512 attr=2, projid32bit=1 2026-03-20T18:00:41.351 INFO:teuthology.orchestra.run.vm03.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-03-20T18:00:41.351 INFO:teuthology.orchestra.run.vm03.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-03-20T18:00:41.351 INFO:teuthology.orchestra.run.vm03.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-03-20T18:00:41.351 INFO:teuthology.orchestra.run.vm03.stdout: = sunit=0 swidth=0 blks 2026-03-20T18:00:41.351 INFO:teuthology.orchestra.run.vm03.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-03-20T18:00:41.351 INFO:teuthology.orchestra.run.vm03.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-03-20T18:00:41.351 INFO:teuthology.orchestra.run.vm03.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-03-20T18:00:41.351 INFO:teuthology.orchestra.run.vm03.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-03-20T18:00:41.355 INFO:teuthology.orchestra.run.vm03.stdout:Discarding blocks...Done. 2026-03-20T18:00:41.358 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@vm03.local -o noatime 2026-03-20T18:00:41.358 DEBUG:teuthology.orchestra.run.vm03:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2026-03-20T18:00:41.429 DEBUG:teuthology.orchestra.run.vm03:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2026-03-20T18:00:41.498 DEBUG:teuthology.orchestra.run.vm03:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2026-03-20T18:00:41.563 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2026-03-20T18:00:41.563 INFO:tasks.ceph:role: osd.6 2026-03-20T18:00:41.563 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@vm03.local 2026-03-20T18:00:41.563 DEBUG:teuthology.orchestra.run.vm03:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-03-20T18:00:41.629 INFO:teuthology.orchestra.run.vm03.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=1310464 blks 2026-03-20T18:00:41.629 INFO:teuthology.orchestra.run.vm03.stdout: = sectsz=512 attr=2, projid32bit=1 2026-03-20T18:00:41.629 INFO:teuthology.orchestra.run.vm03.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-03-20T18:00:41.629 INFO:teuthology.orchestra.run.vm03.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-03-20T18:00:41.629 INFO:teuthology.orchestra.run.vm03.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-03-20T18:00:41.629 INFO:teuthology.orchestra.run.vm03.stdout: = sunit=0 swidth=0 blks 2026-03-20T18:00:41.629 INFO:teuthology.orchestra.run.vm03.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-03-20T18:00:41.629 INFO:teuthology.orchestra.run.vm03.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-03-20T18:00:41.629 INFO:teuthology.orchestra.run.vm03.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-03-20T18:00:41.629 INFO:teuthology.orchestra.run.vm03.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-03-20T18:00:41.634 INFO:teuthology.orchestra.run.vm03.stdout:Discarding blocks...Done. 2026-03-20T18:00:41.636 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@vm03.local -o noatime 2026-03-20T18:00:41.636 DEBUG:teuthology.orchestra.run.vm03:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2026-03-20T18:00:41.705 DEBUG:teuthology.orchestra.run.vm03:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2026-03-20T18:00:41.774 DEBUG:teuthology.orchestra.run.vm03:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2026-03-20T18:00:41.839 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2026-03-20T18:00:41.839 INFO:tasks.ceph:role: osd.7 2026-03-20T18:00:41.839 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@vm03.local 2026-03-20T18:00:41.839 DEBUG:teuthology.orchestra.run.vm03:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-03-20T18:00:41.904 INFO:teuthology.orchestra.run.vm03.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=1310464 blks 2026-03-20T18:00:41.904 INFO:teuthology.orchestra.run.vm03.stdout: = sectsz=512 attr=2, projid32bit=1 2026-03-20T18:00:41.904 INFO:teuthology.orchestra.run.vm03.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-03-20T18:00:41.904 INFO:teuthology.orchestra.run.vm03.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-03-20T18:00:41.904 INFO:teuthology.orchestra.run.vm03.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-03-20T18:00:41.904 INFO:teuthology.orchestra.run.vm03.stdout: = sunit=0 swidth=0 blks 2026-03-20T18:00:41.904 INFO:teuthology.orchestra.run.vm03.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-03-20T18:00:41.904 INFO:teuthology.orchestra.run.vm03.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-03-20T18:00:41.904 INFO:teuthology.orchestra.run.vm03.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-03-20T18:00:41.904 INFO:teuthology.orchestra.run.vm03.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-03-20T18:00:41.908 INFO:teuthology.orchestra.run.vm03.stdout:Discarding blocks...Done. 2026-03-20T18:00:41.911 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@vm03.local -o noatime 2026-03-20T18:00:41.911 DEBUG:teuthology.orchestra.run.vm03:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2026-03-20T18:00:41.982 DEBUG:teuthology.orchestra.run.vm03:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2026-03-20T18:00:42.055 DEBUG:teuthology.orchestra.run.vm03:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:42.140 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:42.140+0000 7f49e4a0e900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2026-03-20T18:00:42.140 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:42.140+0000 7f49e4a0e900 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2026-03-20T18:00:42.140 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:42.140+0000 7f49e4a0e900 -1 bdev(0x559b39ce9800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2026-03-20T18:00:42.140 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:42.140+0000 7f49e4a0e900 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2026-03-20T18:00:42.804 DEBUG:teuthology.orchestra.run.vm03:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2026-03-20T18:00:42.831 DEBUG:teuthology.orchestra.run.vm03:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:42.919 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:42.919+0000 7fd77de97900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2026-03-20T18:00:42.919 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:42.919+0000 7fd77de97900 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2026-03-20T18:00:42.919 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:42.919+0000 7fd77de97900 -1 bdev(0x55ccd32ab800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2026-03-20T18:00:42.919 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:42.919+0000 7fd77de97900 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2026-03-20T18:00:43.606 DEBUG:teuthology.orchestra.run.vm03:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2026-03-20T18:00:43.633 DEBUG:teuthology.orchestra.run.vm03:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:43.717 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:43.718+0000 7f51e7daa900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2026-03-20T18:00:43.718 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:43.718+0000 7f51e7daa900 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2026-03-20T18:00:43.718 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:43.718+0000 7f51e7daa900 -1 bdev(0x565057a2f800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2026-03-20T18:00:43.718 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:43.718+0000 7f51e7daa900 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2026-03-20T18:00:44.432 DEBUG:teuthology.orchestra.run.vm03:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2026-03-20T18:00:44.457 DEBUG:teuthology.orchestra.run.vm03:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:44.536 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:44.536+0000 7f0de2b60900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2026-03-20T18:00:44.536 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:44.536+0000 7f0de2b60900 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2026-03-20T18:00:44.536 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:44.536+0000 7f0de2b60900 -1 bdev(0x555f908d5800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2026-03-20T18:00:44.536 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:00:44.536+0000 7f0de2b60900 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2026-03-20T18:00:45.207 DEBUG:teuthology.orchestra.run.vm03:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2026-03-20T18:00:45.273 INFO:tasks.ceph:Reading keys from all nodes... 2026-03-20T18:00:45.273 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:45.273 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2026-03-20T18:00:45.300 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:45.300 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-03-20T18:00:45.367 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:45.367 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-03-20T18:00:45.435 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:45.436 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-03-20T18:00:45.503 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:45.503 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-03-20T18:00:45.568 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:45.568 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-03-20T18:00:45.592 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:45.592 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2026-03-20T18:00:45.655 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:45.655 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2026-03-20T18:00:45.723 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:45.723 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2026-03-20T18:00:45.787 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:45.787 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2026-03-20T18:00:45.851 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:45.851 DEBUG:teuthology.orchestra.run.vm01:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-03-20T18:00:45.865 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:45.865 DEBUG:teuthology.orchestra.run.vm03:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2026-03-20T18:00:45.906 INFO:tasks.ceph:Adding keys to all mons... 2026-03-20T18:00:45.906 DEBUG:teuthology.orchestra.run.vm01:> sudo tee -a /etc/ceph/ceph.keyring 2026-03-20T18:00:45.908 DEBUG:teuthology.orchestra.run.vm03:> sudo tee -a /etc/ceph/ceph.keyring 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout:[mgr.y] 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout: key = AQBEi71pAGpIGhAAUFOBfwjpVt5y/TyUAAd8Eg== 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout:[osd.0] 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout: key = AQBFi71pLo9TLRAAHmGp2JMzUoQJn5Kf37p61g== 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout:[osd.1] 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout: key = AQBGi71pDr7BIxAAPA1mX+bOJg/HqOCkT242iA== 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout:[osd.2] 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout: key = AQBHi71pIQaQGxAAkRUEGmHju1Xdezun638dTQ== 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout:[osd.3] 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout: key = AQBIi71p3eqjDhAAMCA2D6554rH5KPsiTdGUMQ== 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout:[mgr.x] 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout: key = AQBEi71pDsFsHRAApxCxmt5Go2hKD/Y5l1xzFw== 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout:[osd.4] 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout: key = AQBKi71pHplvCBAAP4TTjpElVId7ph3JOO13kA== 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout:[osd.5] 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout: key = AQBKi71pXwncNhAApeXjTzT/7kYW+zuUj7Pzdg== 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout:[osd.6] 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout: key = AQBLi71p/93dKhAAcEmSw4P1hIxfMstBuEuvHg== 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout:[osd.7] 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout: key = AQBMi71pnPkLIBAAWNxkNOLfluYwQCzqG4emIg== 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout:[client.0] 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout: key = AQBEi71pYS4NIBAALf/p26ScfvuLjH5odQQL2Q== 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout:[client.1] 2026-03-20T18:00:45.951 INFO:teuthology.orchestra.run.vm01.stdout: key = AQBEi71p1ScEIxAA7GOvKKYbNnxCDR2bmfB9PA== 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout:[mgr.y] 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout: key = AQBEi71pAGpIGhAAUFOBfwjpVt5y/TyUAAd8Eg== 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout:[osd.0] 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout: key = AQBFi71pLo9TLRAAHmGp2JMzUoQJn5Kf37p61g== 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout:[osd.1] 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout: key = AQBGi71pDr7BIxAAPA1mX+bOJg/HqOCkT242iA== 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout:[osd.2] 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout: key = AQBHi71pIQaQGxAAkRUEGmHju1Xdezun638dTQ== 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout:[osd.3] 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout: key = AQBIi71p3eqjDhAAMCA2D6554rH5KPsiTdGUMQ== 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout:[mgr.x] 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout: key = AQBEi71pDsFsHRAApxCxmt5Go2hKD/Y5l1xzFw== 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout:[osd.4] 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout: key = AQBKi71pHplvCBAAP4TTjpElVId7ph3JOO13kA== 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout:[osd.5] 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout: key = AQBKi71pXwncNhAApeXjTzT/7kYW+zuUj7Pzdg== 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout:[osd.6] 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout: key = AQBLi71p/93dKhAAcEmSw4P1hIxfMstBuEuvHg== 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout:[osd.7] 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout: key = AQBMi71pnPkLIBAAWNxkNOLfluYwQCzqG4emIg== 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout:[client.0] 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout: key = AQBEi71pYS4NIBAALf/p26ScfvuLjH5odQQL2Q== 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout:[client.1] 2026-03-20T18:00:45.976 INFO:teuthology.orchestra.run.vm03.stdout: key = AQBEi71p1ScEIxAA7GOvKKYbNnxCDR2bmfB9PA== 2026-03-20T18:00:45.977 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-03-20T18:00:45.994 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-03-20T18:00:46.063 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.065 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.110 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.148 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.192 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.193 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.278 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.280 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.323 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-03-20T18:00:46.364 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-03-20T18:00:46.411 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.413 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.465 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.467 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.515 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.517 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.562 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.564 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-20T18:00:46.610 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-03-20T18:00:46.612 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-03-20T18:00:46.666 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-03-20T18:00:46.668 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-03-20T18:00:46.716 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-03-20T18:00:46.716 DEBUG:teuthology.orchestra.run.vm01:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-03-20T18:00:46.738 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-03-20T18:00:46.834 DEBUG:teuthology.orchestra.run.vm01:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-03-20T18:00:46.864 DEBUG:teuthology.orchestra.run.vm01:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2026-03-20T18:00:46.930 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-03-20T18:00:47.027 DEBUG:teuthology.orchestra.run.vm01:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2026-03-20T18:00:47.054 DEBUG:teuthology.orchestra.run.vm03:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2026-03-20T18:00:47.082 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-03-20T18:00:47.178 DEBUG:teuthology.orchestra.run.vm03:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2026-03-20T18:00:47.206 DEBUG:teuthology.orchestra.run.vm01:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:47.208 DEBUG:teuthology.orchestra.run.vm03:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-03-20T18:00:47.264 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-03-20T18:00:47.264 INFO:tasks.ceph.mon.a:Restarting daemon 2026-03-20T18:00:47.265 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-03-20T18:00:47.266 INFO:tasks.ceph.mon.a:Started 2026-03-20T18:00:47.266 INFO:tasks.ceph.mon.c:Restarting daemon 2026-03-20T18:00:47.266 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2026-03-20T18:00:47.268 INFO:tasks.ceph.mon.c:Started 2026-03-20T18:00:47.268 INFO:tasks.ceph.mon.b:Restarting daemon 2026-03-20T18:00:47.268 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2026-03-20T18:00:47.306 INFO:tasks.ceph.mon.b:Started 2026-03-20T18:00:47.307 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-03-20T18:00:47.307 INFO:tasks.ceph.mgr.y:Restarting daemon 2026-03-20T18:00:47.307 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2026-03-20T18:00:47.308 INFO:tasks.ceph.mgr.y:Started 2026-03-20T18:00:47.308 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-03-20T18:00:47.308 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-03-20T18:00:47.309 INFO:tasks.ceph.mgr.x:Started 2026-03-20T18:00:47.309 DEBUG:tasks.ceph:set 0 configs 2026-03-20T18:00:47.309 DEBUG:teuthology.orchestra.run.vm01:> sudo ceph --cluster ceph config dump 2026-03-20T18:00:52.621 INFO:teuthology.orchestra.run.vm01.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-03-20T18:00:52.630 INFO:tasks.ceph:Setting crush tunables to default 2026-03-20T18:00:52.630 DEBUG:teuthology.orchestra.run.vm01:> sudo ceph --cluster ceph osd crush tunables default 2026-03-20T18:00:52.747 INFO:teuthology.orchestra.run.vm01.stderr:adjusted tunables profile to default 2026-03-20T18:00:52.759 INFO:tasks.ceph:check_enable_crimson: False 2026-03-20T18:00:52.759 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-03-20T18:00:52.759 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:52.759 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-03-20T18:00:52.784 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:52.784 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-03-20T18:00:52.852 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:52.852 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-03-20T18:00:52.917 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:00:52.917 DEBUG:teuthology.orchestra.run.vm01:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-03-20T18:00:52.984 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:52.984 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2026-03-20T18:00:53.010 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:53.010 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2026-03-20T18:00:53.077 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:53.077 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2026-03-20T18:00:53.144 DEBUG:teuthology.orchestra.run.vm03:> set -ex 2026-03-20T18:00:53.144 DEBUG:teuthology.orchestra.run.vm03:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2026-03-20T18:00:53.209 DEBUG:teuthology.orchestra.run.vm03:> sudo ceph --cluster ceph osd new f2ab36a2-add8-4864-8cfe-2810e0820fd5 0 2026-03-20T18:00:53.376 INFO:teuthology.orchestra.run.vm03.stdout:0 2026-03-20T18:00:53.386 DEBUG:teuthology.orchestra.run.vm03:> sudo ceph --cluster ceph osd new 3600e1fa-5f59-472e-a0ee-7c7b01234887 1 2026-03-20T18:00:53.418 INFO:tasks.ceph.mgr.x.vm03.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-03-20T18:00:53.418 INFO:tasks.ceph.mgr.x.vm03.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-03-20T18:00:53.418 INFO:tasks.ceph.mgr.x.vm03.stderr: from numpy import show_config as show_numpy_config 2026-03-20T18:00:53.521 INFO:teuthology.orchestra.run.vm03.stdout:1 2026-03-20T18:00:53.531 DEBUG:teuthology.orchestra.run.vm03:> sudo ceph --cluster ceph osd new 759f7c09-a954-43c6-ade8-4c29790a2858 2 2026-03-20T18:00:53.691 INFO:teuthology.orchestra.run.vm03.stdout:2 2026-03-20T18:00:53.701 DEBUG:teuthology.orchestra.run.vm03:> sudo ceph --cluster ceph osd new 5e2a60ed-a259-4bc0-9c29-8d732b028b7d 3 2026-03-20T18:00:53.823 INFO:teuthology.orchestra.run.vm03.stdout:3 2026-03-20T18:00:53.833 DEBUG:teuthology.orchestra.run.vm03:> sudo ceph --cluster ceph osd new 5292c7ad-3d60-4cf3-a3a8-9f388b8fe5d8 4 2026-03-20T18:00:53.951 INFO:teuthology.orchestra.run.vm03.stdout:4 2026-03-20T18:00:53.961 DEBUG:teuthology.orchestra.run.vm03:> sudo ceph --cluster ceph osd new ee1ac2fe-8686-4daa-97d4-f58dd6c37f01 5 2026-03-20T18:00:54.081 INFO:teuthology.orchestra.run.vm03.stdout:5 2026-03-20T18:00:54.091 DEBUG:teuthology.orchestra.run.vm03:> sudo ceph --cluster ceph osd new 517992db-a893-45cf-8813-ea2f4d55c17b 6 2026-03-20T18:00:54.213 INFO:tasks.ceph.mgr.y.vm01.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-03-20T18:00:54.213 INFO:tasks.ceph.mgr.y.vm01.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-03-20T18:00:54.213 INFO:tasks.ceph.mgr.y.vm01.stderr: from numpy import show_config as show_numpy_config 2026-03-20T18:00:54.218 INFO:teuthology.orchestra.run.vm03.stdout:6 2026-03-20T18:00:54.228 DEBUG:teuthology.orchestra.run.vm03:> sudo ceph --cluster ceph osd new 44045000-ab9a-462c-9907-ba4520820a17 7 2026-03-20T18:00:54.346 INFO:teuthology.orchestra.run.vm03.stdout:7 2026-03-20T18:00:54.356 INFO:tasks.ceph.osd.0:Restarting daemon 2026-03-20T18:00:54.356 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-03-20T18:00:54.358 INFO:tasks.ceph.osd.0:Started 2026-03-20T18:00:54.359 INFO:tasks.ceph.osd.1:Restarting daemon 2026-03-20T18:00:54.359 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-03-20T18:00:54.361 INFO:tasks.ceph.osd.1:Started 2026-03-20T18:00:54.361 INFO:tasks.ceph.osd.2:Restarting daemon 2026-03-20T18:00:54.361 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-03-20T18:00:54.363 INFO:tasks.ceph.osd.2:Started 2026-03-20T18:00:54.363 INFO:tasks.ceph.osd.3:Restarting daemon 2026-03-20T18:00:54.363 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-03-20T18:00:54.366 INFO:tasks.ceph.osd.3:Started 2026-03-20T18:00:54.366 INFO:tasks.ceph.osd.4:Restarting daemon 2026-03-20T18:00:54.366 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2026-03-20T18:00:54.367 INFO:tasks.ceph.osd.4:Started 2026-03-20T18:00:54.367 INFO:tasks.ceph.osd.5:Restarting daemon 2026-03-20T18:00:54.368 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2026-03-20T18:00:54.370 INFO:tasks.ceph.osd.5:Started 2026-03-20T18:00:54.370 INFO:tasks.ceph.osd.6:Restarting daemon 2026-03-20T18:00:54.370 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2026-03-20T18:00:54.372 INFO:tasks.ceph.osd.6:Started 2026-03-20T18:00:54.372 INFO:tasks.ceph.osd.7:Restarting daemon 2026-03-20T18:00:54.372 DEBUG:teuthology.orchestra.run.vm03:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2026-03-20T18:00:54.375 INFO:tasks.ceph.osd.7:Started 2026-03-20T18:00:54.375 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-20T18:00:54.532 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:00:54.532 INFO:teuthology.orchestra.run.vm01.stdout:{"epoch":10,"fsid":"e5ff0a4c-c523-496b-bc3f-e531ec13e9f0","created":"2026-03-20T18:00:52.565173+0000","modified":"2026-03-20T18:00:54.344450+0000","last_up_change":"0.000000","last_in_change":"2026-03-20T18:00:54.344450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f2ab36a2-add8-4864-8cfe-2810e0820fd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3600e1fa-5f59-472e-a0ee-7c7b01234887","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"759f7c09-a954-43c6-ade8-4c29790a2858","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5e2a60ed-a259-4bc0-9c29-8d732b028b7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5292c7ad-3d60-4cf3-a3a8-9f388b8fe5d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee1ac2fe-8686-4daa-97d4-f58dd6c37f01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"517992db-a893-45cf-8813-ea2f4d55c17b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"44045000-ab9a-462c-9907-ba4520820a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-20T18:00:54.541 INFO:tasks.ceph.osd.4.vm03.stderr:2026-03-20T18:00:54.541+0000 7fc5fd4d9900 -1 Falling back to public interface 2026-03-20T18:00:54.542 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-03-20T18:00:54.542 INFO:tasks.ceph:Waiting for OSDs to come up 2026-03-20T18:00:54.544 INFO:tasks.ceph.osd.0.vm01.stderr:2026-03-20T18:00:54.544+0000 7f7deb065900 -1 Falling back to public interface 2026-03-20T18:00:54.546 INFO:tasks.ceph.osd.1.vm01.stderr:2026-03-20T18:00:54.547+0000 7f2d79c91900 -1 Falling back to public interface 2026-03-20T18:00:54.557 INFO:tasks.ceph.osd.2.vm01.stderr:2026-03-20T18:00:54.558+0000 7fd7aee0d900 -1 Falling back to public interface 2026-03-20T18:00:54.563 INFO:tasks.ceph.osd.3.vm01.stderr:2026-03-20T18:00:54.564+0000 7fcc1f057900 -1 Falling back to public interface 2026-03-20T18:00:54.570 INFO:tasks.ceph.osd.7.vm03.stderr:2026-03-20T18:00:54.571+0000 7fab9f0f0900 -1 Falling back to public interface 2026-03-20T18:00:54.572 INFO:tasks.ceph.osd.6.vm03.stderr:2026-03-20T18:00:54.573+0000 7f6e5d80c900 -1 Falling back to public interface 2026-03-20T18:00:54.580 INFO:tasks.ceph.osd.5.vm03.stderr:2026-03-20T18:00:54.581+0000 7fbb3fcf0900 -1 Falling back to public interface 2026-03-20T18:00:54.940 INFO:tasks.ceph.osd.0.vm01.stderr:2026-03-20T18:00:54.941+0000 7f7deb065900 -1 osd.0 0 log_to_monitors true 2026-03-20T18:00:54.957 INFO:tasks.ceph.osd.4.vm03.stderr:2026-03-20T18:00:54.957+0000 7fc5fd4d9900 -1 osd.4 0 log_to_monitors true 2026-03-20T18:00:54.994 INFO:tasks.ceph.osd.2.vm01.stderr:2026-03-20T18:00:54.991+0000 7fd7aee0d900 -1 osd.2 0 log_to_monitors true 2026-03-20T18:00:55.002 INFO:tasks.ceph.osd.1.vm01.stderr:2026-03-20T18:00:55.003+0000 7f2d79c91900 -1 osd.1 0 log_to_monitors true 2026-03-20T18:00:55.016 INFO:tasks.ceph.osd.5.vm03.stderr:2026-03-20T18:00:55.015+0000 7fbb3fcf0900 -1 osd.5 0 log_to_monitors true 2026-03-20T18:00:55.025 INFO:tasks.ceph.osd.6.vm03.stderr:2026-03-20T18:00:55.025+0000 7f6e5d80c900 -1 osd.6 0 log_to_monitors true 2026-03-20T18:00:55.041 INFO:tasks.ceph.osd.3.vm01.stderr:2026-03-20T18:00:55.042+0000 7fcc1f057900 -1 osd.3 0 log_to_monitors true 2026-03-20T18:00:55.067 INFO:tasks.ceph.osd.7.vm03.stderr:2026-03-20T18:00:55.067+0000 7fab9f0f0900 -1 osd.7 0 log_to_monitors true 2026-03-20T18:00:55.346 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-03-20T18:00:55.447 INFO:teuthology.misc.health.vm01.stdout: 2026-03-20T18:00:55.447 INFO:teuthology.misc.health.vm01.stdout:{"epoch":10,"fsid":"e5ff0a4c-c523-496b-bc3f-e531ec13e9f0","created":"2026-03-20T18:00:52.565173+0000","modified":"2026-03-20T18:00:54.344450+0000","last_up_change":"0.000000","last_in_change":"2026-03-20T18:00:54.344450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f2ab36a2-add8-4864-8cfe-2810e0820fd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3600e1fa-5f59-472e-a0ee-7c7b01234887","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"759f7c09-a954-43c6-ade8-4c29790a2858","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5e2a60ed-a259-4bc0-9c29-8d732b028b7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5292c7ad-3d60-4cf3-a3a8-9f388b8fe5d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee1ac2fe-8686-4daa-97d4-f58dd6c37f01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"517992db-a893-45cf-8813-ea2f4d55c17b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"44045000-ab9a-462c-9907-ba4520820a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-20T18:00:55.453 DEBUG:teuthology.misc:0 of 8 OSDs are up 2026-03-20T18:00:56.104 INFO:tasks.ceph.mgr.x.vm03.stderr:2026-03-20T18:00:56.104+0000 7f01452fc640 -1 mgr.server handle_report got status from non-daemon mon.c 2026-03-20T18:00:56.104 INFO:tasks.ceph.mgr.x.vm03.stderr:2026-03-20T18:00:56.105+0000 7f01452fc640 -1 mgr.server handle_report got status from non-daemon mon.b 2026-03-20T18:00:56.595 INFO:tasks.ceph.osd.5.vm03.stderr:2026-03-20T18:00:56.596+0000 7fbb3b45e640 -1 osd.5 0 waiting for initial osdmap 2026-03-20T18:00:56.595 INFO:tasks.ceph.osd.4.vm03.stderr:2026-03-20T18:00:56.596+0000 7fc5f8c47640 -1 osd.4 0 waiting for initial osdmap 2026-03-20T18:00:56.595 INFO:tasks.ceph.osd.6.vm03.stderr:2026-03-20T18:00:56.596+0000 7f6e58777640 -1 osd.6 0 waiting for initial osdmap 2026-03-20T18:00:56.596 INFO:tasks.ceph.osd.0.vm01.stderr:2026-03-20T18:00:56.597+0000 7f7de67d1640 -1 osd.0 0 waiting for initial osdmap 2026-03-20T18:00:56.597 INFO:tasks.ceph.osd.2.vm01.stderr:2026-03-20T18:00:56.599+0000 7fd7a9ca1640 -1 osd.2 0 waiting for initial osdmap 2026-03-20T18:00:56.598 INFO:tasks.ceph.osd.3.vm01.stderr:2026-03-20T18:00:56.599+0000 7fcc1a7c3640 -1 osd.3 0 waiting for initial osdmap 2026-03-20T18:00:56.598 INFO:tasks.ceph.osd.1.vm01.stderr:2026-03-20T18:00:56.599+0000 7f2d753fd640 -1 osd.1 0 waiting for initial osdmap 2026-03-20T18:00:56.600 INFO:tasks.ceph.osd.4.vm03.stderr:2026-03-20T18:00:56.600+0000 7fc5f425e640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-20T18:00:56.600 INFO:tasks.ceph.osd.6.vm03.stderr:2026-03-20T18:00:56.600+0000 7f6e53d8e640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-20T18:00:56.600 INFO:tasks.ceph.osd.7.vm03.stderr:2026-03-20T18:00:56.601+0000 7fab9a85e640 -1 osd.7 0 waiting for initial osdmap 2026-03-20T18:00:56.601 INFO:tasks.ceph.osd.5.vm03.stderr:2026-03-20T18:00:56.601+0000 7fbb36a75640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-20T18:00:56.601 INFO:tasks.ceph.osd.0.vm01.stderr:2026-03-20T18:00:56.602+0000 7f7de1de8640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-20T18:00:56.602 INFO:tasks.ceph.osd.1.vm01.stderr:2026-03-20T18:00:56.603+0000 7f2d70a14640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-20T18:00:56.605 INFO:tasks.ceph.osd.3.vm01.stderr:2026-03-20T18:00:56.606+0000 7fcc15dda640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-20T18:00:56.605 INFO:tasks.ceph.osd.2.vm01.stderr:2026-03-20T18:00:56.607+0000 7fd7a52b8640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-20T18:00:56.606 INFO:tasks.ceph.osd.7.vm03.stderr:2026-03-20T18:00:56.606+0000 7fab95e75640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-20T18:01:02.257 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-03-20T18:01:02.467 INFO:teuthology.misc.health.vm01.stdout: 2026-03-20T18:01:02.467 INFO:teuthology.misc.health.vm01.stdout:{"epoch":16,"fsid":"e5ff0a4c-c523-496b-bc3f-e531ec13e9f0","created":"2026-03-20T18:00:52.565173+0000","modified":"2026-03-20T18:01:01.630666+0000","last_up_change":"2026-03-20T18:00:57.595014+0000","last_in_change":"2026-03-20T18:00:54.344450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-20T18:00:59.111312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f2ab36a2-add8-4864-8cfe-2810e0820fd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6800","nonce":346016127},{"type":"v1","addr":"192.168.123.101:6801","nonce":346016127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6802","nonce":346016127},{"type":"v1","addr":"192.168.123.101:6803","nonce":346016127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6806","nonce":346016127},{"type":"v1","addr":"192.168.123.101:6807","nonce":346016127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6804","nonce":346016127},{"type":"v1","addr":"192.168.123.101:6805","nonce":346016127}]},"public_addr":"192.168.123.101:6801/346016127","cluster_addr":"192.168.123.101:6803/346016127","heartbeat_back_addr":"192.168.123.101:6807/346016127","heartbeat_front_addr":"192.168.123.101:6805/346016127","state":["exists","up"]},{"osd":1,"uuid":"3600e1fa-5f59-472e-a0ee-7c7b01234887","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6808","nonce":3038254668},{"type":"v1","addr":"192.168.123.101:6809","nonce":3038254668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6810","nonce":3038254668},{"type":"v1","addr":"192.168.123.101:6811","nonce":3038254668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6814","nonce":3038254668},{"type":"v1","addr":"192.168.123.101:6815","nonce":3038254668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6812","nonce":3038254668},{"type":"v1","addr":"192.168.123.101:6813","nonce":3038254668}]},"public_addr":"192.168.123.101:6809/3038254668","cluster_addr":"192.168.123.101:6811/3038254668","heartbeat_back_addr":"192.168.123.101:6815/3038254668","heartbeat_front_addr":"192.168.123.101:6813/3038254668","state":["exists","up"]},{"osd":2,"uuid":"759f7c09-a954-43c6-ade8-4c29790a2858","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6816","nonce":26512671},{"type":"v1","addr":"192.168.123.101:6817","nonce":26512671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6818","nonce":26512671},{"type":"v1","addr":"192.168.123.101:6819","nonce":26512671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6822","nonce":26512671},{"type":"v1","addr":"192.168.123.101:6823","nonce":26512671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6820","nonce":26512671},{"type":"v1","addr":"192.168.123.101:6821","nonce":26512671}]},"public_addr":"192.168.123.101:6817/26512671","cluster_addr":"192.168.123.101:6819/26512671","heartbeat_back_addr":"192.168.123.101:6823/26512671","heartbeat_front_addr":"192.168.123.101:6821/26512671","state":["exists","up"]},{"osd":3,"uuid":"5e2a60ed-a259-4bc0-9c29-8d732b028b7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6824","nonce":3765595403},{"type":"v1","addr":"192.168.123.101:6825","nonce":3765595403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6826","nonce":3765595403},{"type":"v1","addr":"192.168.123.101:6827","nonce":3765595403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6830","nonce":3765595403},{"type":"v1","addr":"192.168.123.101:6831","nonce":3765595403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6828","nonce":3765595403},{"type":"v1","addr":"192.168.123.101:6829","nonce":3765595403}]},"public_addr":"192.168.123.101:6825/3765595403","cluster_addr":"192.168.123.101:6827/3765595403","heartbeat_back_addr":"192.168.123.101:6831/3765595403","heartbeat_front_addr":"192.168.123.101:6829/3765595403","state":["exists","up"]},{"osd":4,"uuid":"5292c7ad-3d60-4cf3-a3a8-9f388b8fe5d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6800","nonce":1046952930},{"type":"v1","addr":"192.168.123.103:6801","nonce":1046952930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6802","nonce":1046952930},{"type":"v1","addr":"192.168.123.103:6803","nonce":1046952930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6806","nonce":1046952930},{"type":"v1","addr":"192.168.123.103:6807","nonce":1046952930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6804","nonce":1046952930},{"type":"v1","addr":"192.168.123.103:6805","nonce":1046952930}]},"public_addr":"192.168.123.103:6801/1046952930","cluster_addr":"192.168.123.103:6803/1046952930","heartbeat_back_addr":"192.168.123.103:6807/1046952930","heartbeat_front_addr":"192.168.123.103:6805/1046952930","state":["exists","up"]},{"osd":5,"uuid":"ee1ac2fe-8686-4daa-97d4-f58dd6c37f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6824","nonce":852198165},{"type":"v1","addr":"192.168.123.103:6825","nonce":852198165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6826","nonce":852198165},{"type":"v1","addr":"192.168.123.103:6827","nonce":852198165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6830","nonce":852198165},{"type":"v1","addr":"192.168.123.103:6831","nonce":852198165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6828","nonce":852198165},{"type":"v1","addr":"192.168.123.103:6829","nonce":852198165}]},"public_addr":"192.168.123.103:6825/852198165","cluster_addr":"192.168.123.103:6827/852198165","heartbeat_back_addr":"192.168.123.103:6831/852198165","heartbeat_front_addr":"192.168.123.103:6829/852198165","state":["exists","up"]},{"osd":6,"uuid":"517992db-a893-45cf-8813-ea2f4d55c17b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6815","nonce":3686837765},{"type":"v1","addr":"192.168.123.103:6816","nonce":3686837765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6817","nonce":3686837765},{"type":"v1","addr":"192.168.123.103:6818","nonce":3686837765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6821","nonce":3686837765},{"type":"v1","addr":"192.168.123.103:6822","nonce":3686837765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6819","nonce":3686837765},{"type":"v1","addr":"192.168.123.103:6820","nonce":3686837765}]},"public_addr":"192.168.123.103:6816/3686837765","cluster_addr":"192.168.123.103:6818/3686837765","heartbeat_back_addr":"192.168.123.103:6822/3686837765","heartbeat_front_addr":"192.168.123.103:6820/3686837765","state":["exists","up"]},{"osd":7,"uuid":"44045000-ab9a-462c-9907-ba4520820a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6808","nonce":3362858182},{"type":"v1","addr":"192.168.123.103:6809","nonce":3362858182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6810","nonce":3362858182},{"type":"v1","addr":"192.168.123.103:6811","nonce":3362858182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6814","nonce":3362858182},{"type":"v1","addr":"192.168.123.103:6823","nonce":3362858182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6812","nonce":3362858182},{"type":"v1","addr":"192.168.123.103:6813","nonce":3362858182}]},"public_addr":"192.168.123.103:6809/3362858182","cluster_addr":"192.168.123.103:6811/3362858182","heartbeat_back_addr":"192.168.123.103:6823/3362858182","heartbeat_front_addr":"192.168.123.103:6813/3362858182","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-20T18:01:02.477 DEBUG:teuthology.misc:8 of 8 OSDs are up 2026-03-20T18:01:02.477 INFO:tasks.ceph:Creating RBD pool 2026-03-20T18:01:02.477 DEBUG:teuthology.orchestra.run.vm01:> sudo ceph --cluster ceph osd pool create rbd 8 2026-03-20T18:01:03.648 INFO:teuthology.orchestra.run.vm01.stderr:pool 'rbd' created 2026-03-20T18:01:03.661 DEBUG:teuthology.orchestra.run.vm01:> rbd --cluster ceph pool init rbd 2026-03-20T18:01:03.698 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:03.698 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:06.675 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-03-20T18:01:06.676 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-03-20T18:01:06.676 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-03-20T18:01:06.931 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:06.945 INFO:teuthology.orchestra.run.vm01.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-03-20T18:01:06.945 INFO:tasks.ceph_manager:config epoch is 1 2026-03-20T18:01:06.945 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-03-20T18:01:06.945 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-03-20T18:01:06.945 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-03-20T18:01:07.193 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:07.204 INFO:teuthology.orchestra.run.vm01.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6832","nonce":3695960761},{"type":"v1","addr":"192.168.123.103:6833","nonce":3695960761}]},"active_addr":"192.168.123.103:6833/3695960761","active_change":"2026-03-20T18:00:55.087769+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[{"gid":4122,"name":"y","mgr_features":4544132024016699391,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self certificates generated and signed by cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.5","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"how frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self certificates generated and signed by cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.5","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"how frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"192.168.123.103:0","nonce":3907038612}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"192.168.123.103:0","nonce":3532013191}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"192.168.123.103:0","nonce":893075886}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"192.168.123.103:0","nonce":93223512}]}]} 2026-03-20T18:01:07.206 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-03-20T18:01:07.206 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-03-20T18:01:07.206 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-20T18:01:07.414 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:07.414 INFO:teuthology.orchestra.run.vm01.stdout:{"epoch":20,"fsid":"e5ff0a4c-c523-496b-bc3f-e531ec13e9f0","created":"2026-03-20T18:00:52.565173+0000","modified":"2026-03-20T18:01:06.659343+0000","last_up_change":"2026-03-20T18:00:57.595014+0000","last_in_change":"2026-03-20T18:00:54.344450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-20T18:00:59.111312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-03-20T18:01:02.691338+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f2ab36a2-add8-4864-8cfe-2810e0820fd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6800","nonce":346016127},{"type":"v1","addr":"192.168.123.101:6801","nonce":346016127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6802","nonce":346016127},{"type":"v1","addr":"192.168.123.101:6803","nonce":346016127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6806","nonce":346016127},{"type":"v1","addr":"192.168.123.101:6807","nonce":346016127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6804","nonce":346016127},{"type":"v1","addr":"192.168.123.101:6805","nonce":346016127}]},"public_addr":"192.168.123.101:6801/346016127","cluster_addr":"192.168.123.101:6803/346016127","heartbeat_back_addr":"192.168.123.101:6807/346016127","heartbeat_front_addr":"192.168.123.101:6805/346016127","state":["exists","up"]},{"osd":1,"uuid":"3600e1fa-5f59-472e-a0ee-7c7b01234887","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6808","nonce":3038254668},{"type":"v1","addr":"192.168.123.101:6809","nonce":3038254668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6810","nonce":3038254668},{"type":"v1","addr":"192.168.123.101:6811","nonce":3038254668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6814","nonce":3038254668},{"type":"v1","addr":"192.168.123.101:6815","nonce":3038254668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6812","nonce":3038254668},{"type":"v1","addr":"192.168.123.101:6813","nonce":3038254668}]},"public_addr":"192.168.123.101:6809/3038254668","cluster_addr":"192.168.123.101:6811/3038254668","heartbeat_back_addr":"192.168.123.101:6815/3038254668","heartbeat_front_addr":"192.168.123.101:6813/3038254668","state":["exists","up"]},{"osd":2,"uuid":"759f7c09-a954-43c6-ade8-4c29790a2858","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6816","nonce":26512671},{"type":"v1","addr":"192.168.123.101:6817","nonce":26512671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6818","nonce":26512671},{"type":"v1","addr":"192.168.123.101:6819","nonce":26512671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6822","nonce":26512671},{"type":"v1","addr":"192.168.123.101:6823","nonce":26512671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6820","nonce":26512671},{"type":"v1","addr":"192.168.123.101:6821","nonce":26512671}]},"public_addr":"192.168.123.101:6817/26512671","cluster_addr":"192.168.123.101:6819/26512671","heartbeat_back_addr":"192.168.123.101:6823/26512671","heartbeat_front_addr":"192.168.123.101:6821/26512671","state":["exists","up"]},{"osd":3,"uuid":"5e2a60ed-a259-4bc0-9c29-8d732b028b7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6824","nonce":3765595403},{"type":"v1","addr":"192.168.123.101:6825","nonce":3765595403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6826","nonce":3765595403},{"type":"v1","addr":"192.168.123.101:6827","nonce":3765595403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6830","nonce":3765595403},{"type":"v1","addr":"192.168.123.101:6831","nonce":3765595403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6828","nonce":3765595403},{"type":"v1","addr":"192.168.123.101:6829","nonce":3765595403}]},"public_addr":"192.168.123.101:6825/3765595403","cluster_addr":"192.168.123.101:6827/3765595403","heartbeat_back_addr":"192.168.123.101:6831/3765595403","heartbeat_front_addr":"192.168.123.101:6829/3765595403","state":["exists","up"]},{"osd":4,"uuid":"5292c7ad-3d60-4cf3-a3a8-9f388b8fe5d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6800","nonce":1046952930},{"type":"v1","addr":"192.168.123.103:6801","nonce":1046952930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6802","nonce":1046952930},{"type":"v1","addr":"192.168.123.103:6803","nonce":1046952930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6806","nonce":1046952930},{"type":"v1","addr":"192.168.123.103:6807","nonce":1046952930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6804","nonce":1046952930},{"type":"v1","addr":"192.168.123.103:6805","nonce":1046952930}]},"public_addr":"192.168.123.103:6801/1046952930","cluster_addr":"192.168.123.103:6803/1046952930","heartbeat_back_addr":"192.168.123.103:6807/1046952930","heartbeat_front_addr":"192.168.123.103:6805/1046952930","state":["exists","up"]},{"osd":5,"uuid":"ee1ac2fe-8686-4daa-97d4-f58dd6c37f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6824","nonce":852198165},{"type":"v1","addr":"192.168.123.103:6825","nonce":852198165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6826","nonce":852198165},{"type":"v1","addr":"192.168.123.103:6827","nonce":852198165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6830","nonce":852198165},{"type":"v1","addr":"192.168.123.103:6831","nonce":852198165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6828","nonce":852198165},{"type":"v1","addr":"192.168.123.103:6829","nonce":852198165}]},"public_addr":"192.168.123.103:6825/852198165","cluster_addr":"192.168.123.103:6827/852198165","heartbeat_back_addr":"192.168.123.103:6831/852198165","heartbeat_front_addr":"192.168.123.103:6829/852198165","state":["exists","up"]},{"osd":6,"uuid":"517992db-a893-45cf-8813-ea2f4d55c17b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6815","nonce":3686837765},{"type":"v1","addr":"192.168.123.103:6816","nonce":3686837765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6817","nonce":3686837765},{"type":"v1","addr":"192.168.123.103:6818","nonce":3686837765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6821","nonce":3686837765},{"type":"v1","addr":"192.168.123.103:6822","nonce":3686837765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6819","nonce":3686837765},{"type":"v1","addr":"192.168.123.103:6820","nonce":3686837765}]},"public_addr":"192.168.123.103:6816/3686837765","cluster_addr":"192.168.123.103:6818/3686837765","heartbeat_back_addr":"192.168.123.103:6822/3686837765","heartbeat_front_addr":"192.168.123.103:6820/3686837765","state":["exists","up"]},{"osd":7,"uuid":"44045000-ab9a-462c-9907-ba4520820a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6808","nonce":3362858182},{"type":"v1","addr":"192.168.123.103:6809","nonce":3362858182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6810","nonce":3362858182},{"type":"v1","addr":"192.168.123.103:6811","nonce":3362858182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6814","nonce":3362858182},{"type":"v1","addr":"192.168.123.103:6823","nonce":3362858182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6812","nonce":3362858182},{"type":"v1","addr":"192.168.123.103:6813","nonce":3362858182}]},"public_addr":"192.168.123.103:6809/3362858182","cluster_addr":"192.168.123.103:6811/3362858182","heartbeat_back_addr":"192.168.123.103:6823/3362858182","heartbeat_front_addr":"192.168.123.103:6813/3362858182","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-20T18:01:07.425 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-03-20T18:01:07.425 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-20T18:01:07.627 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:07.627 INFO:teuthology.orchestra.run.vm01.stdout:{"epoch":20,"fsid":"e5ff0a4c-c523-496b-bc3f-e531ec13e9f0","created":"2026-03-20T18:00:52.565173+0000","modified":"2026-03-20T18:01:06.659343+0000","last_up_change":"2026-03-20T18:00:57.595014+0000","last_in_change":"2026-03-20T18:00:54.344450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-20T18:00:59.111312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-03-20T18:01:02.691338+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f2ab36a2-add8-4864-8cfe-2810e0820fd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6800","nonce":346016127},{"type":"v1","addr":"192.168.123.101:6801","nonce":346016127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6802","nonce":346016127},{"type":"v1","addr":"192.168.123.101:6803","nonce":346016127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6806","nonce":346016127},{"type":"v1","addr":"192.168.123.101:6807","nonce":346016127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6804","nonce":346016127},{"type":"v1","addr":"192.168.123.101:6805","nonce":346016127}]},"public_addr":"192.168.123.101:6801/346016127","cluster_addr":"192.168.123.101:6803/346016127","heartbeat_back_addr":"192.168.123.101:6807/346016127","heartbeat_front_addr":"192.168.123.101:6805/346016127","state":["exists","up"]},{"osd":1,"uuid":"3600e1fa-5f59-472e-a0ee-7c7b01234887","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6808","nonce":3038254668},{"type":"v1","addr":"192.168.123.101:6809","nonce":3038254668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6810","nonce":3038254668},{"type":"v1","addr":"192.168.123.101:6811","nonce":3038254668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6814","nonce":3038254668},{"type":"v1","addr":"192.168.123.101:6815","nonce":3038254668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6812","nonce":3038254668},{"type":"v1","addr":"192.168.123.101:6813","nonce":3038254668}]},"public_addr":"192.168.123.101:6809/3038254668","cluster_addr":"192.168.123.101:6811/3038254668","heartbeat_back_addr":"192.168.123.101:6815/3038254668","heartbeat_front_addr":"192.168.123.101:6813/3038254668","state":["exists","up"]},{"osd":2,"uuid":"759f7c09-a954-43c6-ade8-4c29790a2858","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6816","nonce":26512671},{"type":"v1","addr":"192.168.123.101:6817","nonce":26512671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6818","nonce":26512671},{"type":"v1","addr":"192.168.123.101:6819","nonce":26512671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6822","nonce":26512671},{"type":"v1","addr":"192.168.123.101:6823","nonce":26512671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6820","nonce":26512671},{"type":"v1","addr":"192.168.123.101:6821","nonce":26512671}]},"public_addr":"192.168.123.101:6817/26512671","cluster_addr":"192.168.123.101:6819/26512671","heartbeat_back_addr":"192.168.123.101:6823/26512671","heartbeat_front_addr":"192.168.123.101:6821/26512671","state":["exists","up"]},{"osd":3,"uuid":"5e2a60ed-a259-4bc0-9c29-8d732b028b7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6824","nonce":3765595403},{"type":"v1","addr":"192.168.123.101:6825","nonce":3765595403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6826","nonce":3765595403},{"type":"v1","addr":"192.168.123.101:6827","nonce":3765595403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6830","nonce":3765595403},{"type":"v1","addr":"192.168.123.101:6831","nonce":3765595403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.101:6828","nonce":3765595403},{"type":"v1","addr":"192.168.123.101:6829","nonce":3765595403}]},"public_addr":"192.168.123.101:6825/3765595403","cluster_addr":"192.168.123.101:6827/3765595403","heartbeat_back_addr":"192.168.123.101:6831/3765595403","heartbeat_front_addr":"192.168.123.101:6829/3765595403","state":["exists","up"]},{"osd":4,"uuid":"5292c7ad-3d60-4cf3-a3a8-9f388b8fe5d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6800","nonce":1046952930},{"type":"v1","addr":"192.168.123.103:6801","nonce":1046952930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6802","nonce":1046952930},{"type":"v1","addr":"192.168.123.103:6803","nonce":1046952930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6806","nonce":1046952930},{"type":"v1","addr":"192.168.123.103:6807","nonce":1046952930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6804","nonce":1046952930},{"type":"v1","addr":"192.168.123.103:6805","nonce":1046952930}]},"public_addr":"192.168.123.103:6801/1046952930","cluster_addr":"192.168.123.103:6803/1046952930","heartbeat_back_addr":"192.168.123.103:6807/1046952930","heartbeat_front_addr":"192.168.123.103:6805/1046952930","state":["exists","up"]},{"osd":5,"uuid":"ee1ac2fe-8686-4daa-97d4-f58dd6c37f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6824","nonce":852198165},{"type":"v1","addr":"192.168.123.103:6825","nonce":852198165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6826","nonce":852198165},{"type":"v1","addr":"192.168.123.103:6827","nonce":852198165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6830","nonce":852198165},{"type":"v1","addr":"192.168.123.103:6831","nonce":852198165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6828","nonce":852198165},{"type":"v1","addr":"192.168.123.103:6829","nonce":852198165}]},"public_addr":"192.168.123.103:6825/852198165","cluster_addr":"192.168.123.103:6827/852198165","heartbeat_back_addr":"192.168.123.103:6831/852198165","heartbeat_front_addr":"192.168.123.103:6829/852198165","state":["exists","up"]},{"osd":6,"uuid":"517992db-a893-45cf-8813-ea2f4d55c17b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6815","nonce":3686837765},{"type":"v1","addr":"192.168.123.103:6816","nonce":3686837765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6817","nonce":3686837765},{"type":"v1","addr":"192.168.123.103:6818","nonce":3686837765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6821","nonce":3686837765},{"type":"v1","addr":"192.168.123.103:6822","nonce":3686837765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6819","nonce":3686837765},{"type":"v1","addr":"192.168.123.103:6820","nonce":3686837765}]},"public_addr":"192.168.123.103:6816/3686837765","cluster_addr":"192.168.123.103:6818/3686837765","heartbeat_back_addr":"192.168.123.103:6822/3686837765","heartbeat_front_addr":"192.168.123.103:6820/3686837765","state":["exists","up"]},{"osd":7,"uuid":"44045000-ab9a-462c-9907-ba4520820a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6808","nonce":3362858182},{"type":"v1","addr":"192.168.123.103:6809","nonce":3362858182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6810","nonce":3362858182},{"type":"v1","addr":"192.168.123.103:6811","nonce":3362858182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6814","nonce":3362858182},{"type":"v1","addr":"192.168.123.103:6823","nonce":3362858182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.103:6812","nonce":3362858182},{"type":"v1","addr":"192.168.123.103:6813","nonce":3362858182}]},"public_addr":"192.168.123.103:6809/3362858182","cluster_addr":"192.168.123.103:6811/3362858182","heartbeat_back_addr":"192.168.123.103:6823/3362858182","heartbeat_front_addr":"192.168.123.103:6813/3362858182","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-20T18:01:07.640 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-20T18:01:07.640 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-20T18:01:07.640 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-20T18:01:07.640 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-20T18:01:07.640 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2026-03-20T18:01:07.640 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2026-03-20T18:01:07.640 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2026-03-20T18:01:07.640 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2026-03-20T18:01:07.829 INFO:teuthology.orchestra.run.vm01.stdout:55834574851 2026-03-20T18:01:07.829 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2026-03-20T18:01:07.853 INFO:teuthology.orchestra.run.vm01.stdout:55834574851 2026-03-20T18:01:07.853 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-20T18:01:07.855 INFO:teuthology.orchestra.run.vm01.stdout:55834574851 2026-03-20T18:01:07.856 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-20T18:01:07.870 INFO:teuthology.orchestra.run.vm01.stdout:55834574851 2026-03-20T18:01:07.870 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-20T18:01:07.878 INFO:teuthology.orchestra.run.vm01.stdout:55834574851 2026-03-20T18:01:07.878 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2026-03-20T18:01:07.880 INFO:teuthology.orchestra.run.vm01.stdout:55834574851 2026-03-20T18:01:07.880 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2026-03-20T18:01:07.890 INFO:teuthology.orchestra.run.vm01.stdout:55834574852 2026-03-20T18:01:07.890 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2026-03-20T18:01:07.890 INFO:teuthology.orchestra.run.vm01.stdout:55834574851 2026-03-20T18:01:07.890 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-20T18:01:08.186 INFO:teuthology.orchestra.run.vm01.stdout:55834574850 2026-03-20T18:01:08.206 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.6 2026-03-20T18:01:08.269 INFO:teuthology.orchestra.run.vm01.stdout:55834574850 2026-03-20T18:01:08.298 INFO:teuthology.orchestra.run.vm01.stdout:55834574850 2026-03-20T18:01:08.304 INFO:teuthology.orchestra.run.vm01.stdout:55834574850 2026-03-20T18:01:08.307 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.2 2026-03-20T18:01:08.314 INFO:teuthology.orchestra.run.vm01.stdout:55834574850 2026-03-20T18:01:08.317 INFO:teuthology.orchestra.run.vm01.stdout:55834574850 2026-03-20T18:01:08.321 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.3 2026-03-20T18:01:08.327 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.5 2026-03-20T18:01:08.333 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.0 2026-03-20T18:01:08.342 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.1 2026-03-20T18:01:08.345 INFO:teuthology.orchestra.run.vm01.stdout:55834574850 2026-03-20T18:01:08.347 INFO:teuthology.orchestra.run.vm01.stdout:55834574851 2026-03-20T18:01:08.360 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.7 2026-03-20T18:01:08.360 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.4 2026-03-20T18:01:09.206 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2026-03-20T18:01:09.308 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-20T18:01:09.323 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-20T18:01:09.328 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2026-03-20T18:01:09.334 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-20T18:01:09.342 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-20T18:01:09.360 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2026-03-20T18:01:09.360 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2026-03-20T18:01:09.473 INFO:teuthology.orchestra.run.vm01.stdout:55834574850 2026-03-20T18:01:09.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.6 2026-03-20T18:01:09.620 INFO:teuthology.orchestra.run.vm01.stdout:55834574850 2026-03-20T18:01:09.641 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.2 2026-03-20T18:01:09.679 INFO:teuthology.orchestra.run.vm01.stdout:55834574850 2026-03-20T18:01:09.708 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.5 2026-03-20T18:01:09.734 INFO:teuthology.orchestra.run.vm01.stdout:55834574852 2026-03-20T18:01:09.748 INFO:teuthology.orchestra.run.vm01.stdout:55834574852 2026-03-20T18:01:09.757 INFO:teuthology.orchestra.run.vm01.stdout:55834574852 2026-03-20T18:01:09.758 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.0 2026-03-20T18:01:09.758 DEBUG:teuthology.parallel:result is None 2026-03-20T18:01:09.764 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.4 2026-03-20T18:01:09.764 DEBUG:teuthology.parallel:result is None 2026-03-20T18:01:09.779 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.3 2026-03-20T18:01:09.779 DEBUG:teuthology.parallel:result is None 2026-03-20T18:01:09.786 INFO:teuthology.orchestra.run.vm01.stdout:55834574853 2026-03-20T18:01:09.792 INFO:teuthology.orchestra.run.vm01.stdout:55834574852 2026-03-20T18:01:09.801 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.7 2026-03-20T18:01:09.801 DEBUG:teuthology.parallel:result is None 2026-03-20T18:01:09.807 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.1 2026-03-20T18:01:09.807 DEBUG:teuthology.parallel:result is None 2026-03-20T18:01:10.498 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2026-03-20T18:01:10.642 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-20T18:01:10.709 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2026-03-20T18:01:10.726 INFO:teuthology.orchestra.run.vm01.stdout:55834574852 2026-03-20T18:01:10.742 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.6 2026-03-20T18:01:10.742 DEBUG:teuthology.parallel:result is None 2026-03-20T18:01:10.862 INFO:teuthology.orchestra.run.vm01.stdout:55834574852 2026-03-20T18:01:10.877 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.2 2026-03-20T18:01:10.877 DEBUG:teuthology.parallel:result is None 2026-03-20T18:01:10.924 INFO:teuthology.orchestra.run.vm01.stdout:55834574852 2026-03-20T18:01:10.936 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.5 2026-03-20T18:01:10.936 DEBUG:teuthology.parallel:result is None 2026-03-20T18:01:10.936 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-03-20T18:01:10.936 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-20T18:01:11.142 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:11.142 INFO:teuthology.orchestra.run.vm01.stderr:dumped all 2026-03-20T18:01:11.154 INFO:teuthology.orchestra.run.vm01.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2026-03-20T18:01:11.098613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":235,"num_write_kb":4762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217624,"kb_used_data":2768,"kb_used_omap":65,"kb_used_meta":214462,"kb_avail":754757096,"statfs":{"total":773094113280,"available":772871266304,"internally_reserved":0,"allocated":2834432,"data_stored":1726734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":66957,"internal_metadata":219609715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.445272"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:06.947102+0000","last_change":"2026-03-20T18:01:06.947102+0000","last_active":"2026-03-20T18:01:06.947102+0000","last_peered":"2026-03-20T18:01:06.947102+0000","last_clean":"2026-03-20T18:01:06.947102+0000","last_became_active":"2026-03-20T18:01:04.658001+0000","last_became_peered":"2026-03-20T18:01:04.658001+0000","last_unstale":"2026-03-20T18:01:06.947102+0000","last_undegraded":"2026-03-20T18:01:06.947102+0000","last_fullsized":"2026-03-20T18:01:06.947102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T23:17:27.314645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025855300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:06.668546+0000","last_change":"2026-03-20T18:01:06.668546+0000","last_active":"2026-03-20T18:01:06.668546+0000","last_peered":"2026-03-20T18:01:06.668546+0000","last_clean":"2026-03-20T18:01:06.668546+0000","last_became_active":"2026-03-20T18:01:04.662688+0000","last_became_peered":"2026-03-20T18:01:04.662688+0000","last_unstale":"2026-03-20T18:01:06.668546+0000","last_undegraded":"2026-03-20T18:01:06.668546+0000","last_fullsized":"2026-03-20T18:01:06.668546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T18:06:38.494585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051543499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:07.197749+0000","last_change":"2026-03-20T18:01:07.197749+0000","last_active":"2026-03-20T18:01:07.197749+0000","last_peered":"2026-03-20T18:01:07.197749+0000","last_clean":"2026-03-20T18:01:07.197749+0000","last_became_active":"2026-03-20T18:01:04.657874+0000","last_became_peered":"2026-03-20T18:01:04.657874+0000","last_unstale":"2026-03-20T18:01:07.197749+0000","last_undegraded":"2026-03-20T18:01:07.197749+0000","last_fullsized":"2026-03-20T18:01:07.197749+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-22T05:57:52.540307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033217099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:06.668548+0000","last_change":"2026-03-20T18:01:06.668548+0000","last_active":"2026-03-20T18:01:06.668548+0000","last_peered":"2026-03-20T18:01:06.668548+0000","last_clean":"2026-03-20T18:01:06.668548+0000","last_became_active":"2026-03-20T18:01:04.660348+0000","last_became_peered":"2026-03-20T18:01:04.660348+0000","last_unstale":"2026-03-20T18:01:06.668548+0000","last_undegraded":"2026-03-20T18:01:06.668548+0000","last_fullsized":"2026-03-20T18:01:06.668548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T21:51:26.542514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049033699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:06.672562+0000","last_change":"2026-03-20T18:01:06.669825+0000","last_active":"2026-03-20T18:01:06.672562+0000","last_peered":"2026-03-20T18:01:06.672562+0000","last_clean":"2026-03-20T18:01:06.672562+0000","last_became_active":"2026-03-20T18:01:04.659952+0000","last_became_peered":"2026-03-20T18:01:04.659952+0000","last_unstale":"2026-03-20T18:01:06.672562+0000","last_undegraded":"2026-03-20T18:01:06.672562+0000","last_fullsized":"2026-03-20T18:01:06.672562+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-22T01:47:58.801993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080626000000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:06.665611+0000","last_change":"2026-03-20T18:01:06.665611+0000","last_active":"2026-03-20T18:01:06.665611+0000","last_peered":"2026-03-20T18:01:06.665611+0000","last_clean":"2026-03-20T18:01:06.665611+0000","last_became_active":"2026-03-20T18:01:04.659551+0000","last_became_peered":"2026-03-20T18:01:04.659551+0000","last_unstale":"2026-03-20T18:01:06.665611+0000","last_undegraded":"2026-03-20T18:01:06.665611+0000","last_fullsized":"2026-03-20T18:01:06.665611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T21:07:13.535599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036308899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:07.197712+0000","last_change":"2026-03-20T18:01:07.197712+0000","last_active":"2026-03-20T18:01:07.197712+0000","last_peered":"2026-03-20T18:01:07.197712+0000","last_clean":"2026-03-20T18:01:07.197712+0000","last_became_active":"2026-03-20T18:01:04.660123+0000","last_became_peered":"2026-03-20T18:01:04.660123+0000","last_unstale":"2026-03-20T18:01:07.197712+0000","last_undegraded":"2026-03-20T18:01:07.197712+0000","last_fullsized":"2026-03-20T18:01:07.197712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T19:54:58.168165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020570500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:06.669379+0000","last_change":"2026-03-20T18:01:06.669379+0000","last_active":"2026-03-20T18:01:06.669379+0000","last_peered":"2026-03-20T18:01:06.669379+0000","last_clean":"2026-03-20T18:01:06.669379+0000","last_became_active":"2026-03-20T18:01:04.659119+0000","last_became_peered":"2026-03-20T18:01:04.659119+0000","last_unstale":"2026-03-20T18:01:06.669379+0000","last_undegraded":"2026-03-20T18:01:06.669379+0000","last_fullsized":"2026-03-20T18:01:06.669379+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T22:58:08.051731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034964499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'192","reported_seq":249,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:07.197546+0000","last_change":"2026-03-20T18:01:01.038085+0000","last_active":"2026-03-20T18:01:07.197546+0000","last_peered":"2026-03-20T18:01:07.197546+0000","last_clean":"2026-03-20T18:01:07.197546+0000","last_became_active":"2026-03-20T18:01:01.037919+0000","last_became_peered":"2026-03-20T18:01:01.037919+0000","last_unstale":"2026-03-20T18:01:07.197546+0000","last_undegraded":"2026-03-20T18:01:07.197546+0000","last_fullsized":"2026-03-20T18:01:07.197546+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:00:59.618028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:00:59.618028+0000","last_clean_scrub_stamp":"2026-03-20T18:00:59.618028+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":100,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T19:30:28.237251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":233,"num_write_kb":4760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":233,"num_write_kb":4760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":780,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":798720,"data_stored":658613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8772,"internal_metadata":27450812},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27056,"kb_used_data":200,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344784,"statfs":{"total":96636764160,"available":96609058816,"internally_reserved":0,"allocated":204800,"data_stored":68245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8776,"internal_metadata":27450808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27068,"kb_used_data":204,"kb_used_omap":7,"kb_used_meta":26808,"kb_avail":94344772,"statfs":{"total":96636764160,"available":96609046528,"internally_reserved":0,"allocated":208896,"data_stored":68264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8126,"internal_metadata":27451458},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27064,"kb_used_data":200,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344776,"statfs":{"total":96636764160,"available":96609050624,"internally_reserved":0,"allocated":204800,"data_stored":68245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8777,"internal_metadata":27450807},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27056,"kb_used_data":200,"kb_used_omap":6,"kb_used_meta":26809,"kb_avail":94344784,"statfs":{"total":96636764160,"available":96609058816,"internally_reserved":0,"allocated":204800,"data_stored":68245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6175,"internal_metadata":27453409},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27064,"kb_used_data":200,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344776,"statfs":{"total":96636764160,"available":96609050624,"internally_reserved":0,"allocated":204800,"data_stored":68245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8777,"internal_metadata":27450807},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27060,"kb_used_data":204,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344780,"statfs":{"total":96636764160,"available":96609054720,"internally_reserved":0,"allocated":208896,"data_stored":68264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8777,"internal_metadata":27450807},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":780,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":798720,"data_stored":658613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8777,"internal_metadata":27450807},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-03-20T18:01:11.155 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-20T18:01:11.363 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:11.363 INFO:teuthology.orchestra.run.vm01.stderr:dumped all 2026-03-20T18:01:11.376 INFO:teuthology.orchestra.run.vm01.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2026-03-20T18:01:11.098613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":235,"num_write_kb":4762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217624,"kb_used_data":2768,"kb_used_omap":65,"kb_used_meta":214462,"kb_avail":754757096,"statfs":{"total":773094113280,"available":772871266304,"internally_reserved":0,"allocated":2834432,"data_stored":1726734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":66957,"internal_metadata":219609715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.445272"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:06.947102+0000","last_change":"2026-03-20T18:01:06.947102+0000","last_active":"2026-03-20T18:01:06.947102+0000","last_peered":"2026-03-20T18:01:06.947102+0000","last_clean":"2026-03-20T18:01:06.947102+0000","last_became_active":"2026-03-20T18:01:04.658001+0000","last_became_peered":"2026-03-20T18:01:04.658001+0000","last_unstale":"2026-03-20T18:01:06.947102+0000","last_undegraded":"2026-03-20T18:01:06.947102+0000","last_fullsized":"2026-03-20T18:01:06.947102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T23:17:27.314645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025855300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:06.668546+0000","last_change":"2026-03-20T18:01:06.668546+0000","last_active":"2026-03-20T18:01:06.668546+0000","last_peered":"2026-03-20T18:01:06.668546+0000","last_clean":"2026-03-20T18:01:06.668546+0000","last_became_active":"2026-03-20T18:01:04.662688+0000","last_became_peered":"2026-03-20T18:01:04.662688+0000","last_unstale":"2026-03-20T18:01:06.668546+0000","last_undegraded":"2026-03-20T18:01:06.668546+0000","last_fullsized":"2026-03-20T18:01:06.668546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T18:06:38.494585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051543499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:07.197749+0000","last_change":"2026-03-20T18:01:07.197749+0000","last_active":"2026-03-20T18:01:07.197749+0000","last_peered":"2026-03-20T18:01:07.197749+0000","last_clean":"2026-03-20T18:01:07.197749+0000","last_became_active":"2026-03-20T18:01:04.657874+0000","last_became_peered":"2026-03-20T18:01:04.657874+0000","last_unstale":"2026-03-20T18:01:07.197749+0000","last_undegraded":"2026-03-20T18:01:07.197749+0000","last_fullsized":"2026-03-20T18:01:07.197749+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-22T05:57:52.540307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033217099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:06.668548+0000","last_change":"2026-03-20T18:01:06.668548+0000","last_active":"2026-03-20T18:01:06.668548+0000","last_peered":"2026-03-20T18:01:06.668548+0000","last_clean":"2026-03-20T18:01:06.668548+0000","last_became_active":"2026-03-20T18:01:04.660348+0000","last_became_peered":"2026-03-20T18:01:04.660348+0000","last_unstale":"2026-03-20T18:01:06.668548+0000","last_undegraded":"2026-03-20T18:01:06.668548+0000","last_fullsized":"2026-03-20T18:01:06.668548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T21:51:26.542514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049033699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:06.672562+0000","last_change":"2026-03-20T18:01:06.669825+0000","last_active":"2026-03-20T18:01:06.672562+0000","last_peered":"2026-03-20T18:01:06.672562+0000","last_clean":"2026-03-20T18:01:06.672562+0000","last_became_active":"2026-03-20T18:01:04.659952+0000","last_became_peered":"2026-03-20T18:01:04.659952+0000","last_unstale":"2026-03-20T18:01:06.672562+0000","last_undegraded":"2026-03-20T18:01:06.672562+0000","last_fullsized":"2026-03-20T18:01:06.672562+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-22T01:47:58.801993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080626000000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:06.665611+0000","last_change":"2026-03-20T18:01:06.665611+0000","last_active":"2026-03-20T18:01:06.665611+0000","last_peered":"2026-03-20T18:01:06.665611+0000","last_clean":"2026-03-20T18:01:06.665611+0000","last_became_active":"2026-03-20T18:01:04.659551+0000","last_became_peered":"2026-03-20T18:01:04.659551+0000","last_unstale":"2026-03-20T18:01:06.665611+0000","last_undegraded":"2026-03-20T18:01:06.665611+0000","last_fullsized":"2026-03-20T18:01:06.665611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T21:07:13.535599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036308899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:07.197712+0000","last_change":"2026-03-20T18:01:07.197712+0000","last_active":"2026-03-20T18:01:07.197712+0000","last_peered":"2026-03-20T18:01:07.197712+0000","last_clean":"2026-03-20T18:01:07.197712+0000","last_became_active":"2026-03-20T18:01:04.660123+0000","last_became_peered":"2026-03-20T18:01:04.660123+0000","last_unstale":"2026-03-20T18:01:07.197712+0000","last_undegraded":"2026-03-20T18:01:07.197712+0000","last_fullsized":"2026-03-20T18:01:07.197712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T19:54:58.168165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020570500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:06.669379+0000","last_change":"2026-03-20T18:01:06.669379+0000","last_active":"2026-03-20T18:01:06.669379+0000","last_peered":"2026-03-20T18:01:06.669379+0000","last_clean":"2026-03-20T18:01:06.669379+0000","last_became_active":"2026-03-20T18:01:04.659119+0000","last_became_peered":"2026-03-20T18:01:04.659119+0000","last_unstale":"2026-03-20T18:01:06.669379+0000","last_undegraded":"2026-03-20T18:01:06.669379+0000","last_fullsized":"2026-03-20T18:01:06.669379+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:01:03.640216+0000","last_clean_scrub_stamp":"2026-03-20T18:01:03.640216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T22:58:08.051731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034964499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'192","reported_seq":249,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-03-20T18:01:07.197546+0000","last_change":"2026-03-20T18:01:01.038085+0000","last_active":"2026-03-20T18:01:07.197546+0000","last_peered":"2026-03-20T18:01:07.197546+0000","last_clean":"2026-03-20T18:01:07.197546+0000","last_became_active":"2026-03-20T18:01:01.037919+0000","last_became_peered":"2026-03-20T18:01:01.037919+0000","last_unstale":"2026-03-20T18:01:07.197546+0000","last_undegraded":"2026-03-20T18:01:07.197546+0000","last_fullsized":"2026-03-20T18:01:07.197546+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-20T18:00:59.618028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-20T18:00:59.618028+0000","last_clean_scrub_stamp":"2026-03-20T18:00:59.618028+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":100,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-03-21T19:30:28.237251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":233,"num_write_kb":4760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":233,"num_write_kb":4760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":780,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":798720,"data_stored":658613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8772,"internal_metadata":27450812},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27056,"kb_used_data":200,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344784,"statfs":{"total":96636764160,"available":96609058816,"internally_reserved":0,"allocated":204800,"data_stored":68245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8776,"internal_metadata":27450808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27068,"kb_used_data":204,"kb_used_omap":7,"kb_used_meta":26808,"kb_avail":94344772,"statfs":{"total":96636764160,"available":96609046528,"internally_reserved":0,"allocated":208896,"data_stored":68264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8126,"internal_metadata":27451458},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27064,"kb_used_data":200,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344776,"statfs":{"total":96636764160,"available":96609050624,"internally_reserved":0,"allocated":204800,"data_stored":68245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8777,"internal_metadata":27450807},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27056,"kb_used_data":200,"kb_used_omap":6,"kb_used_meta":26809,"kb_avail":94344784,"statfs":{"total":96636764160,"available":96609058816,"internally_reserved":0,"allocated":204800,"data_stored":68245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6175,"internal_metadata":27453409},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27064,"kb_used_data":200,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344776,"statfs":{"total":96636764160,"available":96609050624,"internally_reserved":0,"allocated":204800,"data_stored":68245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8777,"internal_metadata":27450807},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27060,"kb_used_data":204,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344780,"statfs":{"total":96636764160,"available":96609054720,"internally_reserved":0,"allocated":208896,"data_stored":68264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8777,"internal_metadata":27450807},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":780,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":798720,"data_stored":658613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8777,"internal_metadata":27450807},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-03-20T18:01:11.377 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-03-20T18:01:11.377 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-03-20T18:01:11.377 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-03-20T18:01:11.377 DEBUG:teuthology.orchestra.run.vm01:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-03-20T18:01:11.610 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:11.610 INFO:teuthology.orchestra.run.vm01.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-03-20T18:01:11.623 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-03-20T18:01:11.624 INFO:teuthology.run_tasks:Running task openssl_keys... 2026-03-20T18:01:11.627 INFO:teuthology.run_tasks:Running task rgw... 2026-03-20T18:01:11.631 DEBUG:tasks.rgw:config is {'client.0': {'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'client.1': {'valgrind': ['--tool=memcheck', '--max-threads=1024']}} 2026-03-20T18:01:11.631 DEBUG:tasks.rgw:client list is dict_keys(['client.0', 'client.1']) 2026-03-20T18:01:11.631 INFO:tasks.rgw:Creating data pools 2026-03-20T18:01:11.631 DEBUG:tasks.rgw:Obtaining remote for client client.0 2026-03-20T18:01:11.631 DEBUG:teuthology.orchestra.run.vm01:> sudo ceph osd pool create MyZone.rgw.buckets.data 64 64 --cluster ceph 2026-03-20T18:01:12.356 INFO:teuthology.orchestra.run.vm01.stderr:pool 'MyZone.rgw.buckets.data' created 2026-03-20T18:01:12.368 DEBUG:teuthology.orchestra.run.vm01:> sudo ceph osd pool application enable MyZone.rgw.buckets.data rgw --cluster ceph 2026-03-20T18:01:13.366 INFO:teuthology.orchestra.run.vm01.stderr:enabled application 'rgw' on pool 'MyZone.rgw.buckets.data' 2026-03-20T18:01:13.386 DEBUG:teuthology.orchestra.run.vm01:> sudo ceph osd pool create MyZone.rgw.buckets.index 64 64 --cluster ceph 2026-03-20T18:01:14.365 INFO:teuthology.orchestra.run.vm01.stderr:pool 'MyZone.rgw.buckets.index' created 2026-03-20T18:01:14.391 DEBUG:teuthology.orchestra.run.vm01:> sudo ceph osd pool application enable MyZone.rgw.buckets.index rgw --cluster ceph 2026-03-20T18:01:15.368 INFO:teuthology.orchestra.run.vm01.stderr:enabled application 'rgw' on pool 'MyZone.rgw.buckets.index' 2026-03-20T18:01:15.404 DEBUG:tasks.rgw:Obtaining remote for client client.1 2026-03-20T18:01:15.404 DEBUG:teuthology.orchestra.run.vm03:> sudo ceph osd pool create MyZone.rgw.buckets.data 64 64 --cluster ceph 2026-03-20T18:01:15.617 INFO:teuthology.orchestra.run.vm03.stderr:pool 'MyZone.rgw.buckets.data' already exists 2026-03-20T18:01:15.630 DEBUG:teuthology.orchestra.run.vm03:> sudo ceph osd pool application enable MyZone.rgw.buckets.data rgw --cluster ceph 2026-03-20T18:01:16.392 INFO:teuthology.orchestra.run.vm03.stderr:enabled application 'rgw' on pool 'MyZone.rgw.buckets.data' 2026-03-20T18:01:16.406 DEBUG:teuthology.orchestra.run.vm03:> sudo ceph osd pool create MyZone.rgw.buckets.index 64 64 --cluster ceph 2026-03-20T18:01:16.619 INFO:teuthology.orchestra.run.vm03.stderr:pool 'MyZone.rgw.buckets.index' already exists 2026-03-20T18:01:16.630 DEBUG:teuthology.orchestra.run.vm03:> sudo ceph osd pool application enable MyZone.rgw.buckets.index rgw --cluster ceph 2026-03-20T18:01:17.411 INFO:teuthology.orchestra.run.vm03.stderr:enabled application 'rgw' on pool 'MyZone.rgw.buckets.index' 2026-03-20T18:01:17.424 DEBUG:tasks.rgw:Pools created 2026-03-20T18:01:17.424 INFO:tasks.rgw:Creating realm MyRealm 2026-03-20T18:01:17.424 INFO:tasks.util.rgw:rgwadmin: client.0 : ['realm', 'create', '--rgw-realm', 'MyRealm', '--default'] 2026-03-20T18:01:17.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'realm', 'create', '--rgw-realm', 'MyRealm', '--default'] 2026-03-20T18:01:17.424 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph realm create --rgw-realm MyRealm --default 2026-03-20T18:01:17.463 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:17.463 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:17.478 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:17.478+0000 7f4fc7019900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:01:17.478 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:17.478+0000 7f4fc7019900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:01:17.478 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:17.479+0000 7f4f70fe1640 20 reqs_thread_entry: start 2026-03-20T18:01:19.466 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.467+0000 7f4fc7019900 20 creating initial latest_epoch=1 for period=ea3b70a4-f7f1-4109-b0b3-40b836ea4cde 2026-03-20T18:01:19.479 INFO:teuthology.orchestra.run.vm01.stdout:{"id":"184e499c-426a-4683-9ccd-d819b580f6c1","name":"MyRealm","current_period":"ea3b70a4-f7f1-4109-b0b3-40b836ea4cde","epoch":1} 2026-03-20T18:01:19.479 DEBUG:tasks.util.rgw: json result: {'id': '184e499c-426a-4683-9ccd-d819b580f6c1', 'name': 'MyRealm', 'current_period': 'ea3b70a4-f7f1-4109-b0b3-40b836ea4cde', 'epoch': 1} 2026-03-20T18:01:19.479 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'create', '--rgw-realm', 'MyRealm', '--rgw-zonegroup', 'MyZoneGroup', '--master', '--default'] 2026-03-20T18:01:19.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'create', '--rgw-realm', 'MyRealm', '--rgw-zonegroup', 'MyZoneGroup', '--master', '--default'] 2026-03-20T18:01:19.480 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup create --rgw-realm MyRealm --rgw-zonegroup MyZoneGroup --master --default 2026-03-20T18:01:19.557 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:19.557 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:19.571 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.572+0000 7fa2dadcb900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:01:19.571 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.572+0000 7fa2dadcb900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:01:19.571 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.572+0000 7fa27f7fe640 20 reqs_thread_entry: start 2026-03-20T18:01:19.582 INFO:teuthology.orchestra.run.vm01.stdout:{"id":"bef9bd2b-4b23-45c1-9e03-2dec3d64903d","name":"MyZoneGroup","api_name":"MyZoneGroup","is_master":true,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"184e499c-426a-4683-9ccd-d819b580f6c1","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]} 2026-03-20T18:01:19.582 DEBUG:tasks.util.rgw: json result: {'id': 'bef9bd2b-4b23-45c1-9e03-2dec3d64903d', 'name': 'MyZoneGroup', 'api_name': 'MyZoneGroup', 'is_master': True, 'endpoints': [], 'hostnames': [], 'hostnames_s3website': [], 'master_zone': '', 'zones': [], 'placement_targets': [{'name': 'default-placement', 'tags': [], 'storage_classes': []}], 'default_placement': 'default-placement', 'realm_id': '184e499c-426a-4683-9ccd-d819b580f6c1', 'sync_policy': {'groups': []}, 'enabled_features': ['notification_v2', 'resharding']} 2026-03-20T18:01:19.582 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'create', '--rgw-realm', 'MyRealm', '--rgw-zonegroup', 'MyZoneGroup', '--rgw-zone', 'MyZone', '--master', '--default'] 2026-03-20T18:01:19.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'create', '--rgw-realm', 'MyRealm', '--rgw-zonegroup', 'MyZoneGroup', '--rgw-zone', 'MyZone', '--master', '--default'] 2026-03-20T18:01:19.582 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone create --rgw-realm MyRealm --rgw-zonegroup MyZoneGroup --rgw-zone MyZone --master --default 2026-03-20T18:01:19.660 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:19.660 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:19.674 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.675+0000 7f941af26900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:01:19.674 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.675+0000 7f941af26900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:01:19.674 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.675+0000 7f93c5fe3640 20 reqs_thread_entry: start 2026-03-20T18:01:19.676 INFO:teuthology.orchestra.run.vm01.stderr:NOTICE: set zone's realm_id=184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:01:19.700 INFO:teuthology.orchestra.run.vm01.stdout:{"id":"c5d78a84-c400-4c18-a0be-5d12df626549","name":"MyZone","domain_root":"MyZone.rgw.meta:root","control_pool":"MyZone.rgw.control","dedup_pool":"MyZone.rgw.dedup","gc_pool":"MyZone.rgw.log:gc","lc_pool":"MyZone.rgw.log:lc","log_pool":"MyZone.rgw.log","intent_log_pool":"MyZone.rgw.log:intent","usage_log_pool":"MyZone.rgw.log:usage","roles_pool":"MyZone.rgw.meta:roles","reshard_pool":"MyZone.rgw.log:reshard","user_keys_pool":"MyZone.rgw.meta:users.keys","user_email_pool":"MyZone.rgw.meta:users.email","user_swift_pool":"MyZone.rgw.meta:users.swift","user_uid_pool":"MyZone.rgw.meta:users.uid","otp_pool":"MyZone.rgw.otp","notif_pool":"MyZone.rgw.log:notif","topics_pool":"MyZone.rgw.meta:topics","account_pool":"MyZone.rgw.meta:accounts","group_pool":"MyZone.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"MyZone.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"MyZone.rgw.buckets.data"}},"data_extra_pool":"MyZone.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"184e499c-426a-4683-9ccd-d819b580f6c1","restore_pool":"MyZone.rgw.log:restore"} 2026-03-20T18:01:19.700 DEBUG:tasks.util.rgw: json result: {'id': 'c5d78a84-c400-4c18-a0be-5d12df626549', 'name': 'MyZone', 'domain_root': 'MyZone.rgw.meta:root', 'control_pool': 'MyZone.rgw.control', 'dedup_pool': 'MyZone.rgw.dedup', 'gc_pool': 'MyZone.rgw.log:gc', 'lc_pool': 'MyZone.rgw.log:lc', 'log_pool': 'MyZone.rgw.log', 'intent_log_pool': 'MyZone.rgw.log:intent', 'usage_log_pool': 'MyZone.rgw.log:usage', 'roles_pool': 'MyZone.rgw.meta:roles', 'reshard_pool': 'MyZone.rgw.log:reshard', 'user_keys_pool': 'MyZone.rgw.meta:users.keys', 'user_email_pool': 'MyZone.rgw.meta:users.email', 'user_swift_pool': 'MyZone.rgw.meta:users.swift', 'user_uid_pool': 'MyZone.rgw.meta:users.uid', 'otp_pool': 'MyZone.rgw.otp', 'notif_pool': 'MyZone.rgw.log:notif', 'topics_pool': 'MyZone.rgw.meta:topics', 'account_pool': 'MyZone.rgw.meta:accounts', 'group_pool': 'MyZone.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'MyZone.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'MyZone.rgw.buckets.data'}}, 'data_extra_pool': 'MyZone.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '184e499c-426a-4683-9ccd-d819b580f6c1', 'restore_pool': 'MyZone.rgw.log:restore'} 2026-03-20T18:01:19.700 INFO:tasks.util.rgw:rgwadmin: client.0 : ['period', 'update', '--commit', '--rgw-realm', 'MyRealm', '--rgw-zonegroup', 'MyZoneGroup', '--rgw-zone', 'MyZone'] 2026-03-20T18:01:19.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'period', 'update', '--commit', '--rgw-realm', 'MyRealm', '--rgw-zonegroup', 'MyZoneGroup', '--rgw-zone', 'MyZone'] 2026-03-20T18:01:19.700 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph period update --commit --rgw-realm MyRealm --rgw-zonegroup MyZoneGroup --rgw-zone MyZone 2026-03-20T18:01:19.779 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:19.779 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:19.798 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.799+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.799 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.800+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:01:19.799 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.800+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.800 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.801+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:01:19.800 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.801+0000 7f5f12017900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:01:19.800 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.801+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.800 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.801+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:01:19.800 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.801+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.801 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.802+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:01:19.801 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.802+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.801 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.802+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=359 2026-03-20T18:01:19.801 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.802+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.802 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.803+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:01:19.802 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.803+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.803 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.804+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=1070 2026-03-20T18:01:19.803 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.804+0000 7f5f12017900 0 period (ea3b70a4-f7f1-4109-b0b3-40b836ea4cde does not have zone c5d78a84-c400-4c18-a0be-5d12df626549 configured 2026-03-20T18:01:19.803 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.804+0000 7f5f12017900 20 searching for the correct realm 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got realms.184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got default.zonegroup.184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got zonegroup_info.bef9bd2b-4b23-45c1-9e03-2dec3d64903d 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got default.realm 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got period_config.184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got zone_info.c5d78a84-c400-4c18-a0be-5d12df626549 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got default.zone.184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got zone_names.MyZone 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got realms.184e499c-426a-4683-9ccd-d819b580f6c1.control 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got periods.ea3b70a4-f7f1-4109-b0b3-40b836ea4cde.1 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got realms_names.MyRealm 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got zonegroups_names.MyZoneGroup 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 RGWRados::pool_iterate: got periods.ea3b70a4-f7f1-4109-b0b3-40b836ea4cde.latest_epoch 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.812+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.813+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.813+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.813+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.813+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.813+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.813+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.813+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.813+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=359 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=517 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 20 zone MyZone found 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 10 cannot find current period zonegroup using local zonegroup configuration 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 20 zonegroup MyZoneGroup 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 20 rados_obj.operate() r=0 bl.length=187 2026-03-20T18:01:19.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:19.814+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:21.472 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:21.473+0000 7f5f12017900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:01:21.472 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:21.473+0000 7f5f12017900 20 rados->read ofs=0 len=0 2026-03-20T18:01:21.472 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:21.473+0000 7f5f12017900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:01:21.472 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:21.473+0000 7f5f12017900 20 started sync module instance, tier type = 2026-03-20T18:01:21.472 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:21.473+0000 7f5f12017900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:01:23.492 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.493+0000 7f5f12017900 20 add_watcher() i=3 2026-03-20T18:01:23.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.494+0000 7f5f12017900 20 add_watcher() i=6 2026-03-20T18:01:23.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.494+0000 7f5f12017900 20 add_watcher() i=0 2026-03-20T18:01:23.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.494+0000 7f5f12017900 20 add_watcher() i=5 2026-03-20T18:01:23.498 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.499+0000 7f5f12017900 20 add_watcher() i=2 2026-03-20T18:01:23.498 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.499+0000 7f5f12017900 20 add_watcher() i=4 2026-03-20T18:01:23.499 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.500+0000 7f5f12017900 20 add_watcher() i=7 2026-03-20T18:01:23.499 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.500+0000 7f5f12017900 20 add_watcher() i=1 2026-03-20T18:01:23.499 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.500+0000 7f5f12017900 2 all 8 watchers are set, enabling cache 2026-03-20T18:01:23.500 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.501+0000 7f5efe7fc640 5 boost::asio::awaitable, obj_version> > logback_generations::read(const DoutPrefixProvider*):446: oid=data_loggenerations_metadata not found 2026-03-20T18:01:23.500 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.501+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.0 2026-03-20T18:01:23.500 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.501+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.501 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.502+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.0 does not exist 2026-03-20T18:01:23.501 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.502+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.1 2026-03-20T18:01:23.501 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.502+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.502 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.503+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.1 does not exist 2026-03-20T18:01:23.502 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.503+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.2 2026-03-20T18:01:23.502 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.503+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.502 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.504+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.2 does not exist 2026-03-20T18:01:23.502 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.504+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.3 2026-03-20T18:01:23.502 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.504+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.503 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.504+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.3 does not exist 2026-03-20T18:01:23.503 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.504+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.4 2026-03-20T18:01:23.503 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.504+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.503 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.504+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.4 does not exist 2026-03-20T18:01:23.503 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.504+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.5 2026-03-20T18:01:23.503 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.504+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.505+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.5 does not exist 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.505+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.6 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.505+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.505+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.6 does not exist 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.505+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.7 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.505+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.505+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.7 does not exist 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.505+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.8 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.505+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.505+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.8 does not exist 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.505+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.9 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.505+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.504 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.506+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.9 does not exist 2026-03-20T18:01:23.505 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.506+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.10 2026-03-20T18:01:23.505 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.506+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.505 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.506+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.10 does not exist 2026-03-20T18:01:23.505 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.506+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.11 2026-03-20T18:01:23.505 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.506+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.505 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.506+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.11 does not exist 2026-03-20T18:01:23.505 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.506+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.12 2026-03-20T18:01:23.505 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.506+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.505 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.507+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.12 does not exist 2026-03-20T18:01:23.505 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.507+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.13 2026-03-20T18:01:23.505 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.507+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.506 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.507+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.13 does not exist 2026-03-20T18:01:23.506 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.507+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.14 2026-03-20T18:01:23.506 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.507+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.506 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.507+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.14 does not exist 2026-03-20T18:01:23.506 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.507+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.15 2026-03-20T18:01:23.506 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.507+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.506 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.507+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.15 does not exist 2026-03-20T18:01:23.506 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.507+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.16 2026-03-20T18:01:23.506 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.507+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.508+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.16 does not exist 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.508+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.17 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.508+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.508+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.17 does not exist 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.508+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.18 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.508+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.508+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.18 does not exist 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.508+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.19 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.508+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.508+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.19 does not exist 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.508+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.20 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.508+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.509+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.20 does not exist 2026-03-20T18:01:23.507 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.509+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.21 2026-03-20T18:01:23.508 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.509+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.508 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.509+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.21 does not exist 2026-03-20T18:01:23.508 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.509+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.22 2026-03-20T18:01:23.508 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.509+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.508 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.509+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.22 does not exist 2026-03-20T18:01:23.508 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.509+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.23 2026-03-20T18:01:23.508 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.509+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.508 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.509+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.23 does not exist 2026-03-20T18:01:23.508 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.509+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.24 2026-03-20T18:01:23.508 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.509+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.508 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.24 does not exist 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.25 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.25 does not exist 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.26 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.26 does not exist 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.27 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.27 does not exist 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.28 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.28 does not exist 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.29 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.510+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.511+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.29 does not exist 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.511+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.30 2026-03-20T18:01:23.509 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.511+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.510 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.511+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.30 does not exist 2026-03-20T18:01:23.510 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.511+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.31 2026-03-20T18:01:23.510 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.511+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.510 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.511+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.31 does not exist 2026-03-20T18:01:23.510 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.511+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.32 2026-03-20T18:01:23.510 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.511+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.510 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.511+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.32 does not exist 2026-03-20T18:01:23.510 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.511+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.33 2026-03-20T18:01:23.510 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.511+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.511 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.512+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.33 does not exist 2026-03-20T18:01:23.511 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.512+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.34 2026-03-20T18:01:23.511 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.512+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.511 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.512+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.34 does not exist 2026-03-20T18:01:23.511 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.512+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.35 2026-03-20T18:01:23.511 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.512+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.511 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.512+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.35 does not exist 2026-03-20T18:01:23.511 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.512+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.36 2026-03-20T18:01:23.511 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.512+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.511 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.512+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.36 does not exist 2026-03-20T18:01:23.511 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.512+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.37 2026-03-20T18:01:23.511 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.512+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.513+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.37 does not exist 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.513+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.38 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.513+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.513+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.38 does not exist 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.513+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.39 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.513+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.513+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.39 does not exist 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.513+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.40 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.513+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.513+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.40 does not exist 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.513+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.41 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.513+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.514+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.41 does not exist 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.514+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.42 2026-03-20T18:01:23.512 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.514+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.513 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.514+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.42 does not exist 2026-03-20T18:01:23.513 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.514+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.43 2026-03-20T18:01:23.513 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.514+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.513 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.514+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.43 does not exist 2026-03-20T18:01:23.513 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.514+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.44 2026-03-20T18:01:23.513 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.514+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.513 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.514+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.44 does not exist 2026-03-20T18:01:23.513 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.514+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.45 2026-03-20T18:01:23.513 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.514+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.513 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.45 does not exist 2026-03-20T18:01:23.513 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.46 2026-03-20T18:01:23.513 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.514 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.46 does not exist 2026-03-20T18:01:23.514 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.47 2026-03-20T18:01:23.514 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.514 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.47 does not exist 2026-03-20T18:01:23.514 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.48 2026-03-20T18:01:23.514 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.514 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.48 does not exist 2026-03-20T18:01:23.514 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.49 2026-03-20T18:01:23.514 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.514 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.49 does not exist 2026-03-20T18:01:23.514 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.50 2026-03-20T18:01:23.514 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.515+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.516+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.50 does not exist 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.516+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.51 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.516+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.516+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.51 does not exist 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.516+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.52 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.516+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.516+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.52 does not exist 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.516+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.53 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.516+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.516+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.53 does not exist 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.516+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.54 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.516+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.54 does not exist 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.55 2026-03-20T18:01:23.515 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.516 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.55 does not exist 2026-03-20T18:01:23.516 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.56 2026-03-20T18:01:23.516 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.516 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.56 does not exist 2026-03-20T18:01:23.516 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.57 2026-03-20T18:01:23.516 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.516 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.57 does not exist 2026-03-20T18:01:23.516 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.58 2026-03-20T18:01:23.516 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.516 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.58 does not exist 2026-03-20T18:01:23.516 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.59 2026-03-20T18:01:23.516 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.517+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.518+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.59 does not exist 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.518+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.60 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.518+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.518+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.60 does not exist 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.518+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.61 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.518+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.518+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.61 does not exist 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.518+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.62 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.518+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.518+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.62 does not exist 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.518+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.63 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.518+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.63 does not exist 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.64 2026-03-20T18:01:23.517 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.64 does not exist 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.65 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.65 does not exist 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.66 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.66 does not exist 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.67 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.67 does not exist 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.68 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.68 does not exist 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.69 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.519+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.518 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.69 does not exist 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.70 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.70 does not exist 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.71 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.71 does not exist 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.72 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.72 does not exist 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.73 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.73 does not exist 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.74 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.520+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.521+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.74 does not exist 2026-03-20T18:01:23.519 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.521+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.75 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.521+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.521+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.75 does not exist 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.521+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.76 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.521+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.521+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.76 does not exist 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.521+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.77 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.521+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.521+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.77 does not exist 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.521+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.78 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.521+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.522+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.78 does not exist 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.522+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.79 2026-03-20T18:01:23.520 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.522+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.521 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.522+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.79 does not exist 2026-03-20T18:01:23.521 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.522+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.80 2026-03-20T18:01:23.521 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.522+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.521 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.522+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.80 does not exist 2026-03-20T18:01:23.521 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.522+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.81 2026-03-20T18:01:23.521 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.522+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.521 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.522+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.81 does not exist 2026-03-20T18:01:23.521 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.522+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.82 2026-03-20T18:01:23.521 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.522+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.521 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.82 does not exist 2026-03-20T18:01:23.521 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.83 2026-03-20T18:01:23.521 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.522 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.83 does not exist 2026-03-20T18:01:23.522 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.84 2026-03-20T18:01:23.522 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.522 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.84 does not exist 2026-03-20T18:01:23.522 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.85 2026-03-20T18:01:23.522 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.522 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.85 does not exist 2026-03-20T18:01:23.522 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.86 2026-03-20T18:01:23.522 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.522 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.86 does not exist 2026-03-20T18:01:23.522 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.87 2026-03-20T18:01:23.522 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.523+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.87 does not exist 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.88 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.88 does not exist 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.89 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.89 does not exist 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.90 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.90 does not exist 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.91 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.91 does not exist 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.92 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.524+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.525+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.92 does not exist 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.525+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.93 2026-03-20T18:01:23.523 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.525+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.524 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.525+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.93 does not exist 2026-03-20T18:01:23.524 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.525+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.94 2026-03-20T18:01:23.524 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.525+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.524 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.525+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.94 does not exist 2026-03-20T18:01:23.524 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.525+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.95 2026-03-20T18:01:23.524 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.525+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.524 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.525+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.95 does not exist 2026-03-20T18:01:23.524 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.525+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.96 2026-03-20T18:01:23.524 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.525+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.524 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.96 does not exist 2026-03-20T18:01:23.524 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.97 2026-03-20T18:01:23.524 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.525 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.97 does not exist 2026-03-20T18:01:23.525 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.98 2026-03-20T18:01:23.525 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.525 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.98 does not exist 2026-03-20T18:01:23.525 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.99 2026-03-20T18:01:23.525 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.525 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.99 does not exist 2026-03-20T18:01:23.525 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.100 2026-03-20T18:01:23.525 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.525 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.100 does not exist 2026-03-20T18:01:23.525 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.101 2026-03-20T18:01:23.525 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.526+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.527+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.101 does not exist 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.527+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.102 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.527+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.527+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.102 does not exist 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.527+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.103 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.527+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.527+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.103 does not exist 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.527+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.104 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.527+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.527+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.104 does not exist 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.527+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.105 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.527+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.105 does not exist 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.106 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.526 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.106 does not exist 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.107 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.107 does not exist 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.108 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.108 does not exist 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.109 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.109 does not exist 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.110 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.110 does not exist 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.111 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.528+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.111 does not exist 2026-03-20T18:01:23.527 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.112 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.112 does not exist 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.113 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.113 does not exist 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.114 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.114 does not exist 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.115 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.115 does not exist 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.116 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.529+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.116 does not exist 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.117 2026-03-20T18:01:23.528 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.117 does not exist 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.118 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.118 does not exist 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.119 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.119 does not exist 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.120 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5efe7fc640 20 do_open: entering 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.120 does not exist 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5efdffb640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.121 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.121 does not exist 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5efd7fa640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.122 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.530+0000 7f5efd7fa640 20 do_open: entering 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.122 does not exist 2026-03-20T18:01:23.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5f0e782640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.123 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5f0e782640 20 do_open: entering 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.123 does not exist 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5efffff640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.124 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5efffff640 20 do_open: entering 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.124 does not exist 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5eff7fe640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.125 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5eff7fe640 20 do_open: entering 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.125 does not exist 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5efeffd640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.126 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5efeffd640 20 do_open: entering 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.126 does not exist 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5f0fa0b640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):59 probing obj=data_log.127 2026-03-20T18:01:23.530 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.531+0000 7f5f0fa0b640 20 do_open: entering 2026-03-20T18:01:23.531 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.532+0000 7f5efe7fc640 20 boost::asio::awaitable {anonymous}::probe_shard(const DoutPrefixProvider*, neorados::RADOS, const neorados::Object&, const neorados::IOContext&, bool&):78: obj=data_log.127 does not exist 2026-03-20T18:01:23.531 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.532+0000 7f5efe7fc640 20 do_create: entering 2026-03-20T18:01:23.532 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.533+0000 7f5efdffb640 20 do_open: entering 2026-03-20T18:01:23.535 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.536+0000 7f5f12017900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:01:23.535 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:23.536+0000 7f5f12017900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:01:26.492 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.493+0000 7f5f12017900 10 rgw_init_ioctx warning: failed to set recovery_priority on MyZone.rgw.meta 2026-03-20T18:01:26.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.493+0000 7f5f12017900 5 note: GC not initialized 2026-03-20T18:01:26.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.494+0000 7f5ebafe5640 20 reqs_thread_entry: start 2026-03-20T18:01:26.553 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.554+0000 7f5f12017900 20 init_complete bucket index max shards: 11 2026-03-20T18:01:26.553 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.554+0000 7f5f12017900 20 Filter name: none 2026-03-20T18:01:26.553 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.554+0000 7f5eb8fe1640 20 reqs_thread_entry: start 2026-03-20T18:01:26.555 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.556+0000 7f5f12017900 20 fork_period realm id=184e499c-426a-4683-9ccd-d819b580f6c1 period id=ea3b70a4-f7f1-4109-b0b3-40b836ea4cde 2026-03-20T18:01:26.567 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.568+0000 7f5f12017900 20 creating initial latest_epoch=1 for period=184e499c-426a-4683-9ccd-d819b580f6c1:staging 2026-03-20T18:01:26.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.570+0000 7f5f12017900 20 commit_period realm 184e499c-426a-4683-9ccd-d819b580f6c1 period ea3b70a4-f7f1-4109-b0b3-40b836ea4cde 2026-03-20T18:01:26.571 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.572+0000 7f5f12017900 20 creating initial latest_epoch=1 for period=0caded40-e639-4de0-bbbe-6af6ee4fba80 2026-03-20T18:01:26.578 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.579+0000 7f5f12017900 4 Promoted to master zone and committed new period 0caded40-e639-4de0-bbbe-6af6ee4fba80 2026-03-20T18:01:26.583 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.584+0000 7f5f12017900 20 remove_watcher() i=0 2026-03-20T18:01:26.583 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.584+0000 7f5f12017900 2 removed watcher, disabling cache 2026-03-20T18:01:26.583 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.584+0000 7f5f12017900 20 remove_watcher() i=3 2026-03-20T18:01:26.584 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.584+0000 7f5f12017900 20 remove_watcher() i=4 2026-03-20T18:01:26.584 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.585+0000 7f5f12017900 20 remove_watcher() i=2 2026-03-20T18:01:26.584 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.585+0000 7f5f12017900 20 remove_watcher() i=7 2026-03-20T18:01:26.584 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.585+0000 7f5f12017900 20 remove_watcher() i=6 2026-03-20T18:01:26.584 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.585+0000 7f5f12017900 20 remove_watcher() i=5 2026-03-20T18:01:26.584 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.585+0000 7f5f12017900 20 remove_watcher() i=1 2026-03-20T18:01:26.591 INFO:teuthology.orchestra.run.vm01.stdout:{"id":"0caded40-e639-4de0-bbbe-6af6ee4fba80","epoch":1,"predecessor_uuid":"ea3b70a4-f7f1-4109-b0b3-40b836ea4cde","sync_status":[],"period_map":{"id":"0caded40-e639-4de0-bbbe-6af6ee4fba80","zonegroups":[{"id":"bef9bd2b-4b23-45c1-9e03-2dec3d64903d","name":"MyZoneGroup","api_name":"MyZoneGroup","is_master":true,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"c5d78a84-c400-4c18-a0be-5d12df626549","zones":[{"id":"c5d78a84-c400-4c18-a0be-5d12df626549","name":"MyZone","endpoints":[],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"184e499c-426a-4683-9ccd-d819b580f6c1","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"c5d78a84-c400-4c18-a0be-5d12df626549","val":600987272}]},"master_zonegroup":"bef9bd2b-4b23-45c1-9e03-2dec3d64903d","master_zone":"c5d78a84-c400-4c18-a0be-5d12df626549","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"184e499c-426a-4683-9ccd-d819b580f6c1","realm_epoch":2} 2026-03-20T18:01:26.591 DEBUG:tasks.util.rgw: json result: {'id': '0caded40-e639-4de0-bbbe-6af6ee4fba80', 'epoch': 1, 'predecessor_uuid': 'ea3b70a4-f7f1-4109-b0b3-40b836ea4cde', 'sync_status': [], 'period_map': {'id': '0caded40-e639-4de0-bbbe-6af6ee4fba80', 'zonegroups': [{'id': 'bef9bd2b-4b23-45c1-9e03-2dec3d64903d', 'name': 'MyZoneGroup', 'api_name': 'MyZoneGroup', 'is_master': True, 'endpoints': [], 'hostnames': [], 'hostnames_s3website': [], 'master_zone': 'c5d78a84-c400-4c18-a0be-5d12df626549', 'zones': [{'id': 'c5d78a84-c400-4c18-a0be-5d12df626549', 'name': 'MyZone', 'endpoints': [], 'log_meta': False, 'log_data': False, 'bucket_index_max_shards': 11, 'read_only': False, 'tier_type': '', 'sync_from_all': True, 'sync_from': [], 'redirect_zone': '', 'supported_features': ['compress-encrypted', 'notification_v2', 'resharding']}], 'placement_targets': [{'name': 'default-placement', 'tags': [], 'storage_classes': ['STANDARD']}], 'default_placement': 'default-placement', 'realm_id': '184e499c-426a-4683-9ccd-d819b580f6c1', 'sync_policy': {'groups': []}, 'enabled_features': ['notification_v2', 'resharding']}], 'short_zone_ids': [{'key': 'c5d78a84-c400-4c18-a0be-5d12df626549', 'val': 600987272}]}, 'master_zonegroup': 'bef9bd2b-4b23-45c1-9e03-2dec3d64903d', 'master_zone': 'c5d78a84-c400-4c18-a0be-5d12df626549', 'period_config': {'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}, 'bucket_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}, 'anonymous_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}}, 'realm_id': '184e499c-426a-4683-9ccd-d819b580f6c1', 'realm_epoch': 2} 2026-03-20T18:01:26.591 INFO:tasks.rgw:Configuring storage class = FROZEN 2026-03-20T18:01:26.591 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2026-03-20T18:01:26.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2026-03-20T18:01:26.591 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone MyZone --placement-id default-placement --storage-class FROZEN 2026-03-20T18:01:26.671 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:26.671 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:26.687 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.688+0000 7ff798926900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:01:26.687 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.688+0000 7ff798926900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:01:26.687 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.688+0000 7ff742fe5640 20 reqs_thread_entry: start 2026-03-20T18:01:26.696 INFO:teuthology.orchestra.run.vm01.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","STANDARD"]}}] 2026-03-20T18:01:26.696 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'STANDARD']}}] 2026-03-20T18:01:26.697 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2026-03-20T18:01:26.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2026-03-20T18:01:26.697 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone MyZone --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2026-03-20T18:01:26.778 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:26.778 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:26.794 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.795+0000 7f0107f4d900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:01:26.794 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.795+0000 7f0107f4d900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:01:26.794 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.795+0000 7f00b1fe3640 20 reqs_thread_entry: start 2026-03-20T18:01:26.805 INFO:teuthology.orchestra.run.vm01.stdout:{"id":"c5d78a84-c400-4c18-a0be-5d12df626549","name":"MyZone","domain_root":"MyZone.rgw.meta:root","control_pool":"MyZone.rgw.control","dedup_pool":"MyZone.rgw.dedup","gc_pool":"MyZone.rgw.log:gc","lc_pool":"MyZone.rgw.log:lc","log_pool":"MyZone.rgw.log","intent_log_pool":"MyZone.rgw.log:intent","usage_log_pool":"MyZone.rgw.log:usage","roles_pool":"MyZone.rgw.meta:roles","reshard_pool":"MyZone.rgw.log:reshard","user_keys_pool":"MyZone.rgw.meta:users.keys","user_email_pool":"MyZone.rgw.meta:users.email","user_swift_pool":"MyZone.rgw.meta:users.swift","user_uid_pool":"MyZone.rgw.meta:users.uid","otp_pool":"MyZone.rgw.otp","notif_pool":"MyZone.rgw.log:notif","topics_pool":"MyZone.rgw.meta:topics","account_pool":"MyZone.rgw.meta:accounts","group_pool":"MyZone.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"MyZone.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"STANDARD":{"data_pool":"MyZone.rgw.buckets.data"}},"data_extra_pool":"MyZone.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"184e499c-426a-4683-9ccd-d819b580f6c1","restore_pool":"MyZone.rgw.log:restore"} 2026-03-20T18:01:26.805 DEBUG:tasks.util.rgw: json result: {'id': 'c5d78a84-c400-4c18-a0be-5d12df626549', 'name': 'MyZone', 'domain_root': 'MyZone.rgw.meta:root', 'control_pool': 'MyZone.rgw.control', 'dedup_pool': 'MyZone.rgw.dedup', 'gc_pool': 'MyZone.rgw.log:gc', 'lc_pool': 'MyZone.rgw.log:lc', 'log_pool': 'MyZone.rgw.log', 'intent_log_pool': 'MyZone.rgw.log:intent', 'usage_log_pool': 'MyZone.rgw.log:usage', 'roles_pool': 'MyZone.rgw.meta:roles', 'reshard_pool': 'MyZone.rgw.log:reshard', 'user_keys_pool': 'MyZone.rgw.meta:users.keys', 'user_email_pool': 'MyZone.rgw.meta:users.email', 'user_swift_pool': 'MyZone.rgw.meta:users.swift', 'user_uid_pool': 'MyZone.rgw.meta:users.uid', 'otp_pool': 'MyZone.rgw.otp', 'notif_pool': 'MyZone.rgw.log:notif', 'topics_pool': 'MyZone.rgw.meta:topics', 'account_pool': 'MyZone.rgw.meta:accounts', 'group_pool': 'MyZone.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'MyZone.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'STANDARD': {'data_pool': 'MyZone.rgw.buckets.data'}}, 'data_extra_pool': 'MyZone.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '184e499c-426a-4683-9ccd-d819b580f6c1', 'restore_pool': 'MyZone.rgw.log:restore'} 2026-03-20T18:01:26.805 INFO:tasks.rgw:Configuring storage class = LUKEWARM 2026-03-20T18:01:26.805 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2026-03-20T18:01:26.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2026-03-20T18:01:26.806 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone MyZone --placement-id default-placement --storage-class LUKEWARM 2026-03-20T18:01:26.886 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:26.887 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:26.903 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.904+0000 7f4c05e13900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:01:26.903 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.904+0000 7f4c05e13900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:01:26.903 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:26.904+0000 7f4baf7e6640 20 reqs_thread_entry: start 2026-03-20T18:01:26.914 INFO:teuthology.orchestra.run.vm01.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","LUKEWARM","STANDARD"]}}] 2026-03-20T18:01:26.914 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2026-03-20T18:01:26.914 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2026-03-20T18:01:26.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2026-03-20T18:01:26.914 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone MyZone --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2026-03-20T18:01:26.997 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:26.997 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:27.013 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:27.013+0000 7f2b9cc76900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:01:27.013 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:27.013+0000 7f2b9cc76900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:01:27.013 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:01:27.014+0000 7f2b497e2640 20 reqs_thread_entry: start 2026-03-20T18:01:27.024 INFO:teuthology.orchestra.run.vm01.stdout:{"id":"c5d78a84-c400-4c18-a0be-5d12df626549","name":"MyZone","domain_root":"MyZone.rgw.meta:root","control_pool":"MyZone.rgw.control","dedup_pool":"MyZone.rgw.dedup","gc_pool":"MyZone.rgw.log:gc","lc_pool":"MyZone.rgw.log:lc","log_pool":"MyZone.rgw.log","intent_log_pool":"MyZone.rgw.log:intent","usage_log_pool":"MyZone.rgw.log:usage","roles_pool":"MyZone.rgw.meta:roles","reshard_pool":"MyZone.rgw.log:reshard","user_keys_pool":"MyZone.rgw.meta:users.keys","user_email_pool":"MyZone.rgw.meta:users.email","user_swift_pool":"MyZone.rgw.meta:users.swift","user_uid_pool":"MyZone.rgw.meta:users.uid","otp_pool":"MyZone.rgw.otp","notif_pool":"MyZone.rgw.log:notif","topics_pool":"MyZone.rgw.meta:topics","account_pool":"MyZone.rgw.meta:accounts","group_pool":"MyZone.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"MyZone.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"MyZone.rgw.buckets.data"}},"data_extra_pool":"MyZone.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"184e499c-426a-4683-9ccd-d819b580f6c1","restore_pool":"MyZone.rgw.log:restore"} 2026-03-20T18:01:27.024 DEBUG:tasks.util.rgw: json result: {'id': 'c5d78a84-c400-4c18-a0be-5d12df626549', 'name': 'MyZone', 'domain_root': 'MyZone.rgw.meta:root', 'control_pool': 'MyZone.rgw.control', 'dedup_pool': 'MyZone.rgw.dedup', 'gc_pool': 'MyZone.rgw.log:gc', 'lc_pool': 'MyZone.rgw.log:lc', 'log_pool': 'MyZone.rgw.log', 'intent_log_pool': 'MyZone.rgw.log:intent', 'usage_log_pool': 'MyZone.rgw.log:usage', 'roles_pool': 'MyZone.rgw.meta:roles', 'reshard_pool': 'MyZone.rgw.log:reshard', 'user_keys_pool': 'MyZone.rgw.meta:users.keys', 'user_email_pool': 'MyZone.rgw.meta:users.email', 'user_swift_pool': 'MyZone.rgw.meta:users.swift', 'user_uid_pool': 'MyZone.rgw.meta:users.uid', 'otp_pool': 'MyZone.rgw.otp', 'notif_pool': 'MyZone.rgw.log:notif', 'topics_pool': 'MyZone.rgw.meta:topics', 'account_pool': 'MyZone.rgw.meta:accounts', 'group_pool': 'MyZone.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'MyZone.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'MyZone.rgw.buckets.data'}}, 'data_extra_pool': 'MyZone.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '184e499c-426a-4683-9ccd-d819b580f6c1', 'restore_pool': 'MyZone.rgw.log:restore'} 2026-03-20T18:01:27.024 INFO:tasks.rgw:Configuring storage class = FROZEN 2026-03-20T18:01:27.024 INFO:tasks.util.rgw:rgwadmin: client.1 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2026-03-20T18:01:27.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2026-03-20T18:01:27.024 DEBUG:teuthology.orchestra.run.vm03:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster ceph zonegroup placement add --rgw-zone MyZone --placement-id default-placement --storage-class FROZEN 2026-03-20T18:01:27.062 INFO:teuthology.orchestra.run.vm03.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:27.063 INFO:teuthology.orchestra.run.vm03.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:27.075 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:01:27.075+0000 7fbb6c210900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:01:27.075 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:01:27.075+0000 7fbb6c210900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:01:27.075 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:01:27.076+0000 7fbb15fe3640 20 reqs_thread_entry: start 2026-03-20T18:01:27.086 INFO:teuthology.orchestra.run.vm03.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","LUKEWARM","STANDARD"]}}] 2026-03-20T18:01:27.086 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2026-03-20T18:01:27.086 INFO:tasks.util.rgw:rgwadmin: client.1 : ['zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2026-03-20T18:01:27.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2026-03-20T18:01:27.086 DEBUG:teuthology.orchestra.run.vm03:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster ceph zone placement add --rgw-zone MyZone --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2026-03-20T18:01:27.165 INFO:teuthology.orchestra.run.vm03.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:27.165 INFO:teuthology.orchestra.run.vm03.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:27.179 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:01:27.179+0000 7f544d74b900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:01:27.179 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:01:27.179+0000 7f544d74b900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:01:27.179 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:01:27.179+0000 7f53f97e2640 20 reqs_thread_entry: start 2026-03-20T18:01:27.189 INFO:teuthology.orchestra.run.vm03.stdout:{"id":"c5d78a84-c400-4c18-a0be-5d12df626549","name":"MyZone","domain_root":"MyZone.rgw.meta:root","control_pool":"MyZone.rgw.control","dedup_pool":"MyZone.rgw.dedup","gc_pool":"MyZone.rgw.log:gc","lc_pool":"MyZone.rgw.log:lc","log_pool":"MyZone.rgw.log","intent_log_pool":"MyZone.rgw.log:intent","usage_log_pool":"MyZone.rgw.log:usage","roles_pool":"MyZone.rgw.meta:roles","reshard_pool":"MyZone.rgw.log:reshard","user_keys_pool":"MyZone.rgw.meta:users.keys","user_email_pool":"MyZone.rgw.meta:users.email","user_swift_pool":"MyZone.rgw.meta:users.swift","user_uid_pool":"MyZone.rgw.meta:users.uid","otp_pool":"MyZone.rgw.otp","notif_pool":"MyZone.rgw.log:notif","topics_pool":"MyZone.rgw.meta:topics","account_pool":"MyZone.rgw.meta:accounts","group_pool":"MyZone.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"MyZone.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"MyZone.rgw.buckets.data"}},"data_extra_pool":"MyZone.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"184e499c-426a-4683-9ccd-d819b580f6c1","restore_pool":"MyZone.rgw.log:restore"} 2026-03-20T18:01:27.189 DEBUG:tasks.util.rgw: json result: {'id': 'c5d78a84-c400-4c18-a0be-5d12df626549', 'name': 'MyZone', 'domain_root': 'MyZone.rgw.meta:root', 'control_pool': 'MyZone.rgw.control', 'dedup_pool': 'MyZone.rgw.dedup', 'gc_pool': 'MyZone.rgw.log:gc', 'lc_pool': 'MyZone.rgw.log:lc', 'log_pool': 'MyZone.rgw.log', 'intent_log_pool': 'MyZone.rgw.log:intent', 'usage_log_pool': 'MyZone.rgw.log:usage', 'roles_pool': 'MyZone.rgw.meta:roles', 'reshard_pool': 'MyZone.rgw.log:reshard', 'user_keys_pool': 'MyZone.rgw.meta:users.keys', 'user_email_pool': 'MyZone.rgw.meta:users.email', 'user_swift_pool': 'MyZone.rgw.meta:users.swift', 'user_uid_pool': 'MyZone.rgw.meta:users.uid', 'otp_pool': 'MyZone.rgw.otp', 'notif_pool': 'MyZone.rgw.log:notif', 'topics_pool': 'MyZone.rgw.meta:topics', 'account_pool': 'MyZone.rgw.meta:accounts', 'group_pool': 'MyZone.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'MyZone.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'MyZone.rgw.buckets.data'}}, 'data_extra_pool': 'MyZone.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '184e499c-426a-4683-9ccd-d819b580f6c1', 'restore_pool': 'MyZone.rgw.log:restore'} 2026-03-20T18:01:27.189 INFO:tasks.rgw:Configuring storage class = LUKEWARM 2026-03-20T18:01:27.189 INFO:tasks.util.rgw:rgwadmin: client.1 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2026-03-20T18:01:27.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2026-03-20T18:01:27.189 DEBUG:teuthology.orchestra.run.vm03:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster ceph zonegroup placement add --rgw-zone MyZone --placement-id default-placement --storage-class LUKEWARM 2026-03-20T18:01:27.272 INFO:teuthology.orchestra.run.vm03.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:27.272 INFO:teuthology.orchestra.run.vm03.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:27.288 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:01:27.288+0000 7f956c52d900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:01:27.288 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:01:27.288+0000 7f956c52d900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:01:27.288 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:01:27.288+0000 7f95167e4640 20 reqs_thread_entry: start 2026-03-20T18:01:27.296 INFO:teuthology.orchestra.run.vm03.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","LUKEWARM","STANDARD"]}}] 2026-03-20T18:01:27.297 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2026-03-20T18:01:27.297 INFO:tasks.util.rgw:rgwadmin: client.1 : ['zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2026-03-20T18:01:27.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2026-03-20T18:01:27.297 DEBUG:teuthology.orchestra.run.vm03:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster ceph zone placement add --rgw-zone MyZone --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2026-03-20T18:01:27.338 INFO:teuthology.orchestra.run.vm03.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:01:27.338 INFO:teuthology.orchestra.run.vm03.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:01:27.352 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:01:27.352+0000 7f1c3532f900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:01:27.352 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:01:27.352+0000 7f1c3532f900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:01:27.352 INFO:teuthology.orchestra.run.vm03.stderr:2026-03-20T18:01:27.353+0000 7f1bdffe7640 20 reqs_thread_entry: start 2026-03-20T18:01:27.363 INFO:teuthology.orchestra.run.vm03.stdout:{"id":"c5d78a84-c400-4c18-a0be-5d12df626549","name":"MyZone","domain_root":"MyZone.rgw.meta:root","control_pool":"MyZone.rgw.control","dedup_pool":"MyZone.rgw.dedup","gc_pool":"MyZone.rgw.log:gc","lc_pool":"MyZone.rgw.log:lc","log_pool":"MyZone.rgw.log","intent_log_pool":"MyZone.rgw.log:intent","usage_log_pool":"MyZone.rgw.log:usage","roles_pool":"MyZone.rgw.meta:roles","reshard_pool":"MyZone.rgw.log:reshard","user_keys_pool":"MyZone.rgw.meta:users.keys","user_email_pool":"MyZone.rgw.meta:users.email","user_swift_pool":"MyZone.rgw.meta:users.swift","user_uid_pool":"MyZone.rgw.meta:users.uid","otp_pool":"MyZone.rgw.otp","notif_pool":"MyZone.rgw.log:notif","topics_pool":"MyZone.rgw.meta:topics","account_pool":"MyZone.rgw.meta:accounts","group_pool":"MyZone.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"MyZone.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"MyZone.rgw.buckets.data"}},"data_extra_pool":"MyZone.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"184e499c-426a-4683-9ccd-d819b580f6c1","restore_pool":"MyZone.rgw.log:restore"} 2026-03-20T18:01:27.364 DEBUG:tasks.util.rgw: json result: {'id': 'c5d78a84-c400-4c18-a0be-5d12df626549', 'name': 'MyZone', 'domain_root': 'MyZone.rgw.meta:root', 'control_pool': 'MyZone.rgw.control', 'dedup_pool': 'MyZone.rgw.dedup', 'gc_pool': 'MyZone.rgw.log:gc', 'lc_pool': 'MyZone.rgw.log:lc', 'log_pool': 'MyZone.rgw.log', 'intent_log_pool': 'MyZone.rgw.log:intent', 'usage_log_pool': 'MyZone.rgw.log:usage', 'roles_pool': 'MyZone.rgw.meta:roles', 'reshard_pool': 'MyZone.rgw.log:reshard', 'user_keys_pool': 'MyZone.rgw.meta:users.keys', 'user_email_pool': 'MyZone.rgw.meta:users.email', 'user_swift_pool': 'MyZone.rgw.meta:users.swift', 'user_uid_pool': 'MyZone.rgw.meta:users.uid', 'otp_pool': 'MyZone.rgw.otp', 'notif_pool': 'MyZone.rgw.log:notif', 'topics_pool': 'MyZone.rgw.meta:topics', 'account_pool': 'MyZone.rgw.meta:accounts', 'group_pool': 'MyZone.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'MyZone.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'MyZone.rgw.buckets.data'}}, 'data_extra_pool': 'MyZone.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '184e499c-426a-4683-9ccd-d819b580f6c1', 'restore_pool': 'MyZone.rgw.log:restore'} 2026-03-20T18:01:27.364 INFO:tasks.rgw:Starting rgw... 2026-03-20T18:01:27.364 INFO:tasks.rgw:rgw client.0 config is {'valgrind': ['--tool=memcheck', '--max-threads=1024']} 2026-03-20T18:01:27.364 INFO:tasks.rgw:Using beast as radosgw frontend 2026-03-20T18:01:27.364 DEBUG:teuthology.orchestra.run.vm01:> sudo echo -n http://vm01.local:80 | sudo tee /home/ubuntu/cephtest/url_file 2026-03-20T18:01:27.389 INFO:teuthology.orchestra.run.vm01.stdout:http://vm01.local:80 2026-03-20T18:01:27.389 DEBUG:teuthology.orchestra.run.vm01:> sudo chown ceph /home/ubuntu/cephtest/url_file 2026-03-20T18:01:27.451 DEBUG:tasks.ceph_manager:running ceph.client.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/ceph.client.0.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--max-threads=1024'] 2026-03-20T18:01:27.451 INFO:tasks.rgw.client.0:Restarting daemon 2026-03-20T18:01:27.451 DEBUG:teuthology.orchestra.run.vm01:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/ceph.client.0.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2026-03-20T18:01:27.492 INFO:tasks.rgw.client.0:Started 2026-03-20T18:01:27.492 INFO:tasks.rgw:rgw client.1 config is {'valgrind': ['--tool=memcheck', '--max-threads=1024']} 2026-03-20T18:01:27.492 INFO:tasks.rgw:Using beast as radosgw frontend 2026-03-20T18:01:27.492 DEBUG:teuthology.orchestra.run.vm03:> sudo echo -n http://vm03.local:80 | sudo tee /home/ubuntu/cephtest/url_file 2026-03-20T18:01:27.524 INFO:teuthology.orchestra.run.vm03.stdout:http://vm03.local:80 2026-03-20T18:01:27.524 DEBUG:teuthology.orchestra.run.vm03:> sudo chown ceph /home/ubuntu/cephtest/url_file 2026-03-20T18:01:27.588 DEBUG:tasks.ceph_manager:running ceph.client.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/ceph.client.1.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--max-threads=1024'] 2026-03-20T18:01:27.588 INFO:tasks.rgw.client.1:Restarting daemon 2026-03-20T18:01:27.588 DEBUG:teuthology.orchestra.run.vm03:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/ceph.client.1.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast port=80' -n client.1 --cluster ceph -k /etc/ceph/ceph.client.1.keyring --log-file /var/log/ceph/rgw.ceph.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.1.stdout 2>&1 2026-03-20T18:01:27.629 INFO:tasks.rgw.client.1:Started 2026-03-20T18:01:27.629 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://vm01.local:80/ 2026-03-20T18:01:27.629 DEBUG:teuthology.orchestra.run.vm01:> curl http://vm01.local:80/ 2026-03-20T18:01:27.648 INFO:teuthology.orchestra.run.vm01.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2026-03-20T18:01:27.648 INFO:teuthology.orchestra.run.vm01.stderr: Dload Upload Total Spent Left Speed 2026-03-20T18:01:27.649 DEBUG:teuthology.orchestra.run:got remote process result: 7 2026-03-20T18:01:27.649 INFO:teuthology.orchestra.run.vm01.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2026-03-20T18:01:27.649 INFO:teuthology.orchestra.run.vm01.stderr:curl: (7) Failed to connect to vm01.local port 80: Connection refused 2026-03-20T18:01:28.650 DEBUG:teuthology.orchestra.run.vm01:> curl http://vm01.local:80/ 2026-03-20T18:01:28.668 INFO:teuthology.orchestra.run.vm01.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2026-03-20T18:01:28.668 INFO:teuthology.orchestra.run.vm01.stderr: Dload Upload Total Spent Left Speed 2026-03-20T18:01:28.668 INFO:teuthology.orchestra.run.vm01.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2026-03-20T18:01:28.669 INFO:teuthology.orchestra.run.vm01.stderr:curl: (7) Failed to connect to vm01.local port 80: Connection refused 2026-03-20T18:01:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 7 2026-03-20T18:01:30.669 DEBUG:teuthology.orchestra.run.vm01:> curl http://vm01.local:80/ 2026-03-20T18:01:30.689 INFO:teuthology.orchestra.run.vm01.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2026-03-20T18:01:30.690 INFO:teuthology.orchestra.run.vm01.stderr: Dload Upload Total Spent Left Speed 2026-03-20T18:01:30.690 INFO:teuthology.orchestra.run.vm01.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2026-03-20T18:01:30.690 INFO:teuthology.orchestra.run.vm01.stderr:curl: (7) Failed to connect to vm01.local port 80: Connection refused 2026-03-20T18:01:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 7 2026-03-20T18:01:34.690 DEBUG:teuthology.orchestra.run.vm01:> curl http://vm01.local:80/ 2026-03-20T18:01:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 7 2026-03-20T18:01:34.717 INFO:teuthology.orchestra.run.vm01.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2026-03-20T18:01:34.717 INFO:teuthology.orchestra.run.vm01.stderr: Dload Upload Total Spent Left Speed 2026-03-20T18:01:34.717 INFO:teuthology.orchestra.run.vm01.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2026-03-20T18:01:34.717 INFO:teuthology.orchestra.run.vm01.stderr:curl: (7) Failed to connect to vm01.local port 80: Connection refused 2026-03-20T18:01:39.546 INFO:tasks.rgw.client.0.vm01.stdout:2026-03-20T18:01:39.542+0000 136681640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):729 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) in Realm id ( 184e499c-426a-4683-9ccd-d819b580f6c1) has no endpoints! 2026-03-20T18:01:39.551 INFO:tasks.rgw.client.0.vm01.stdout:2026-03-20T18:01:39.542+0000 136681640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):739 ERROR: Cluster is is misconfigured! Zone MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) in Zonegroup MyZoneGroup ( bef9bd2b-4b23-45c1-9e03-2dec3d64903d) in Realm id ( 184e499c-426a-4683-9ccd-d819b580f6c1) has no endpoints! Trimming is impossible. 2026-03-20T18:01:39.551 INFO:tasks.rgw.client.0.vm01.stdout:2026-03-20T18:01:39.543+0000 136681640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):758 ERROR: Cluster is is misconfigured! Refusing to trim. 2026-03-20T18:01:39.552 INFO:tasks.rgw.client.0.vm01.stdout:2026-03-20T18:01:39.544+0000 136681640 -1 rgw rados thread: Bailing out of trim thread! 2026-03-20T18:01:39.778 INFO:tasks.rgw.client.1.vm03.stdout:2026-03-20T18:01:39.761+0000 136681640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):729 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) in Realm id ( 184e499c-426a-4683-9ccd-d819b580f6c1) has no endpoints! 2026-03-20T18:01:39.778 INFO:tasks.rgw.client.1.vm03.stdout:2026-03-20T18:01:39.762+0000 136681640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):739 ERROR: Cluster is is misconfigured! Zone MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) in Zonegroup MyZoneGroup ( bef9bd2b-4b23-45c1-9e03-2dec3d64903d) in Realm id ( 184e499c-426a-4683-9ccd-d819b580f6c1) has no endpoints! Trimming is impossible. 2026-03-20T18:01:39.778 INFO:tasks.rgw.client.1.vm03.stdout:2026-03-20T18:01:39.763+0000 136681640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):758 ERROR: Cluster is is misconfigured! Refusing to trim. 2026-03-20T18:01:39.778 INFO:tasks.rgw.client.1.vm03.stdout:2026-03-20T18:01:39.764+0000 136681640 -1 rgw rados thread: Bailing out of trim thread! 2026-03-20T18:01:42.718 DEBUG:teuthology.orchestra.run.vm01:> curl http://vm01.local:80/ 2026-03-20T18:01:42.739 INFO:teuthology.orchestra.run.vm01.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2026-03-20T18:01:42.739 INFO:teuthology.orchestra.run.vm01.stderr: Dload Upload Total Spent Left Speed 2026-03-20T18:01:42.926 INFO:teuthology.orchestra.run.vm01.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 1000 0 --:--:-- --:--:-- --:--:-- 1000 2026-03-20T18:01:42.926 INFO:teuthology.orchestra.run.vm01.stdout:anonymous 2026-03-20T18:01:42.926 INFO:tasks.rgw:Polling client.1 until it starts accepting connections on http://vm03.local:80/ 2026-03-20T18:01:42.926 DEBUG:teuthology.orchestra.run.vm03:> curl http://vm03.local:80/ 2026-03-20T18:01:42.944 INFO:teuthology.orchestra.run.vm03.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2026-03-20T18:01:42.944 INFO:teuthology.orchestra.run.vm03.stderr: Dload Upload Total Spent Left Speed 2026-03-20T18:01:43.135 INFO:teuthology.orchestra.run.vm03.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 984 0 --:--:-- --:--:-- --:--:-- 984 2026-03-20T18:01:43.136 INFO:teuthology.orchestra.run.vm03.stdout:anonymous 2026-03-20T18:01:43.136 INFO:teuthology.run_tasks:Running task rabbitmq... 2026-03-20T18:01:43.148 DEBUG:tasks.rabbitmq:RabbitMQ config is {'client.0': None} 2026-03-20T18:01:43.148 INFO:tasks.rabbitmq:Installing RabbitMQ... 2026-03-20T18:01:43.149 DEBUG:teuthology.orchestra.run.vm01:> sudo dnf -y install epel-release 2026-03-20T18:01:43.528 INFO:teuthology.orchestra.run.vm01.stdout:Last metadata expiration check: 0:02:01 ago on Fri 20 Mar 2026 05:59:42 PM UTC. 2026-03-20T18:01:43.627 INFO:teuthology.orchestra.run.vm01.stdout:Dependencies resolved. 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout: Package Architecture Version Repository Size 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout:Installing: 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout: epel-release noarch 9-10.el9 epel 19 k 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout:Installing weak dependencies: 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout: epel-next-release noarch 9-10.el9 epel 7.9 k 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout:Transaction Summary 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout:Install 2 Packages 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout:Total download size: 26 k 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout:Installed size: 29 k 2026-03-20T18:01:43.628 INFO:teuthology.orchestra.run.vm01.stdout:Downloading Packages: 2026-03-20T18:01:43.768 INFO:teuthology.orchestra.run.vm01.stdout:(1/2): epel-next-release-9-10.el9.noarch.rpm 904 kB/s | 7.9 kB 00:00 2026-03-20T18:01:43.769 INFO:teuthology.orchestra.run.vm01.stdout:(2/2): epel-release-9-10.el9.noarch.rpm 1.9 MB/s | 19 kB 00:00 2026-03-20T18:01:43.769 INFO:teuthology.orchestra.run.vm01.stdout:-------------------------------------------------------------------------------- 2026-03-20T18:01:43.769 INFO:teuthology.orchestra.run.vm01.stdout:Total 187 kB/s | 26 kB 00:00 2026-03-20T18:01:43.769 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction check 2026-03-20T18:01:43.772 INFO:teuthology.orchestra.run.vm01.stdout:Transaction check succeeded. 2026-03-20T18:01:43.772 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction test 2026-03-20T18:01:43.774 INFO:teuthology.orchestra.run.vm01.stdout:Transaction test succeeded. 2026-03-20T18:01:43.774 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction 2026-03-20T18:01:43.799 INFO:teuthology.orchestra.run.vm01.stdout: Preparing : 1/1 2026-03-20T18:01:43.799 INFO:teuthology.orchestra.run.vm01.stdout: Installing : epel-release-9-10.el9.noarch 1/2 2026-03-20T18:01:43.799 INFO:teuthology.orchestra.run.vm01.stdout:warning: /etc/yum.repos.d/epel-testing.repo created as /etc/yum.repos.d/epel-testing.repo.rpmnew 2026-03-20T18:01:43.799 INFO:teuthology.orchestra.run.vm01.stdout:warning: /etc/yum.repos.d/epel.repo created as /etc/yum.repos.d/epel.repo.rpmnew 2026-03-20T18:01:43.799 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:43.806 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: epel-release-9-10.el9.noarch 1/2 2026-03-20T18:01:43.806 INFO:teuthology.orchestra.run.vm01.stdout:Many EPEL packages require the CodeReady Builder (CRB) repository. 2026-03-20T18:01:43.806 INFO:teuthology.orchestra.run.vm01.stdout:It is recommended that you run /usr/bin/crb enable to enable the CRB repository. 2026-03-20T18:01:43.806 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:43.824 INFO:teuthology.orchestra.run.vm01.stdout: Installing : epel-next-release-9-10.el9.noarch 2/2 2026-03-20T18:01:44.076 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: epel-next-release-9-10.el9.noarch 2/2 2026-03-20T18:01:44.076 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : epel-next-release-9-10.el9.noarch 1/2 2026-03-20T18:01:44.140 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : epel-release-9-10.el9.noarch 2/2 2026-03-20T18:01:44.140 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:44.140 INFO:teuthology.orchestra.run.vm01.stdout:Installed: 2026-03-20T18:01:44.140 INFO:teuthology.orchestra.run.vm01.stdout: epel-next-release-9-10.el9.noarch epel-release-9-10.el9.noarch 2026-03-20T18:01:44.140 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:01:44.140 INFO:teuthology.orchestra.run.vm01.stdout:Complete! 2026-03-20T18:01:44.189 DEBUG:teuthology.orchestra.run.vm01:> curl -s https://packagecloud.io/install/repositories/rabbitmq/erlang/script.rpm.sh | sudo bash 2026-03-20T18:01:44.705 INFO:teuthology.orchestra.run.vm01.stdout:Detected operating system as centos/9. 2026-03-20T18:01:44.705 INFO:teuthology.orchestra.run.vm01.stdout:Checking for curl... 2026-03-20T18:01:44.705 INFO:teuthology.orchestra.run.vm01.stdout:Detected curl... 2026-03-20T18:01:44.705 INFO:teuthology.orchestra.run.vm01.stdout:Downloading repository file: https://packagecloud.io/install/repositories/rabbitmq/erlang/config_file.repo?os=centos&dist=9&source=script 2026-03-20T18:01:45.210 INFO:teuthology.orchestra.run.vm01.stdout:done. 2026-03-20T18:01:45.210 INFO:teuthology.orchestra.run.vm01.stdout:Installing yum-utils... 2026-03-20T18:01:45.850 INFO:teuthology.orchestra.run.vm01.stdout:Extra Packages for Enterprise Linux 9 openh264 8.3 kB/s | 2.5 kB 00:00 2026-03-20T18:01:46.068 INFO:teuthology.orchestra.run.vm01.stdout:Extra Packages for Enterprise Linux 9 - Next - 1.8 MB/s | 259 kB 00:00 2026-03-20T18:01:48.384 INFO:teuthology.orchestra.run.vm01.stdout:rabbitmq_erlang-source 381 B/s | 819 B 00:02 2026-03-20T18:01:48.894 INFO:teuthology.orchestra.run.vm01.stdout:rabbitmq_erlang-source 7.6 kB/s | 3.8 kB 00:00 2026-03-20T18:01:48.932 INFO:teuthology.orchestra.run.vm01.stderr:Importing GPG key 0xDF309A0B: 2026-03-20T18:01:48.932 INFO:teuthology.orchestra.run.vm01.stderr: Userid : "https://packagecloud.io/rabbitmq/erlang (https://packagecloud.io/docs#gpg_signing) " 2026-03-20T18:01:48.932 INFO:teuthology.orchestra.run.vm01.stderr: Fingerprint: 2EBD E413 D3CE 5D35 BCD1 5B7C 71C6 3471 DF30 9A0B 2026-03-20T18:01:48.932 INFO:teuthology.orchestra.run.vm01.stderr: From : https://packagecloud.io/rabbitmq/erlang/gpgkey 2026-03-20T18:01:51.793 INFO:teuthology.orchestra.run.vm01.stdout:rabbitmq_erlang-source 104 B/s | 296 B 00:02 2026-03-20T18:01:52.232 INFO:teuthology.orchestra.run.vm01.stdout:Package yum-utils-4.3.0-26.el9.noarch is already installed. 2026-03-20T18:01:52.259 INFO:teuthology.orchestra.run.vm01.stdout:Dependencies resolved. 2026-03-20T18:01:52.260 INFO:teuthology.orchestra.run.vm01.stdout:Nothing to do. 2026-03-20T18:01:52.260 INFO:teuthology.orchestra.run.vm01.stdout:Complete! 2026-03-20T18:01:52.364 INFO:teuthology.orchestra.run.vm01.stdout:Generating yum cache for rabbitmq_erlang... 2026-03-20T18:01:55.141 INFO:teuthology.orchestra.run.vm01.stderr:Importing GPG key 0xDF309A0B: 2026-03-20T18:01:55.141 INFO:teuthology.orchestra.run.vm01.stderr: Userid : "https://packagecloud.io/rabbitmq/erlang (https://packagecloud.io/docs#gpg_signing) " 2026-03-20T18:01:55.141 INFO:teuthology.orchestra.run.vm01.stderr: Fingerprint: 2EBD E413 D3CE 5D35 BCD1 5B7C 71C6 3471 DF30 9A0B 2026-03-20T18:01:55.141 INFO:teuthology.orchestra.run.vm01.stderr: From : https://packagecloud.io/rabbitmq/erlang/gpgkey 2026-03-20T18:01:58.463 INFO:teuthology.orchestra.run.vm01.stdout:Generating yum cache for rabbitmq_erlang-source... 2026-03-20T18:02:00.605 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:02:00.605 INFO:teuthology.orchestra.run.vm01.stdout:The repository is setup! You can now install packages. 2026-03-20T18:02:00.606 DEBUG:teuthology.orchestra.run.vm01:> sudo dnf -y install erlang 2026-03-20T18:02:01.039 INFO:teuthology.orchestra.run.vm01.stdout:Last metadata expiration check: 0:00:01 ago on Fri 20 Mar 2026 06:02:00 PM UTC. 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout:Dependencies resolved. 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout: Package Architecture Version Repository Size 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout:Installing: 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout: erlang x86_64 26.2.5.4-1.el9 rabbitmq_erlang 21 M 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout:Transaction Summary 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout:Install 1 Package 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout:Total download size: 21 M 2026-03-20T18:02:01.189 INFO:teuthology.orchestra.run.vm01.stdout:Installed size: 35 M 2026-03-20T18:02:01.190 INFO:teuthology.orchestra.run.vm01.stdout:Downloading Packages: 2026-03-20T18:02:02.005 INFO:teuthology.orchestra.run.vm01.stdout:erlang-26.2.5.4-1.el9.x86_64.rpm 26 MB/s | 21 MB 00:00 2026-03-20T18:02:02.006 INFO:teuthology.orchestra.run.vm01.stdout:-------------------------------------------------------------------------------- 2026-03-20T18:02:02.007 INFO:teuthology.orchestra.run.vm01.stdout:Total 25 MB/s | 21 MB 00:00 2026-03-20T18:02:02.007 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction check 2026-03-20T18:02:02.012 INFO:teuthology.orchestra.run.vm01.stdout:Transaction check succeeded. 2026-03-20T18:02:02.012 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction test 2026-03-20T18:02:02.129 INFO:teuthology.orchestra.run.vm01.stdout:Transaction test succeeded. 2026-03-20T18:02:02.129 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction 2026-03-20T18:02:02.645 INFO:teuthology.orchestra.run.vm01.stdout: Preparing : 1/1 2026-03-20T18:02:02.671 INFO:teuthology.orchestra.run.vm01.stdout: Installing : erlang-26.2.5.4-1.el9.x86_64 1/1 2026-03-20T18:02:02.808 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: erlang-26.2.5.4-1.el9.x86_64 1/1 2026-03-20T18:02:02.878 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : erlang-26.2.5.4-1.el9.x86_64 1/1 2026-03-20T18:02:02.878 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:02:02.878 INFO:teuthology.orchestra.run.vm01.stdout:Installed: 2026-03-20T18:02:02.878 INFO:teuthology.orchestra.run.vm01.stdout: erlang-26.2.5.4-1.el9.x86_64 2026-03-20T18:02:02.878 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:02:02.878 INFO:teuthology.orchestra.run.vm01.stdout:Complete! 2026-03-20T18:02:02.918 DEBUG:teuthology.orchestra.run.vm01:> curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash 2026-03-20T18:02:03.441 INFO:teuthology.orchestra.run.vm01.stdout:Detected operating system as centos/9. 2026-03-20T18:02:03.441 INFO:teuthology.orchestra.run.vm01.stdout:Checking for curl... 2026-03-20T18:02:03.441 INFO:teuthology.orchestra.run.vm01.stdout:Detected curl... 2026-03-20T18:02:03.441 INFO:teuthology.orchestra.run.vm01.stdout:Downloading repository file: https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/config_file.repo?os=centos&dist=9&source=script 2026-03-20T18:02:03.958 INFO:teuthology.orchestra.run.vm01.stdout:done. 2026-03-20T18:02:03.958 INFO:teuthology.orchestra.run.vm01.stdout:Installing yum-utils... 2026-03-20T18:02:06.497 INFO:teuthology.orchestra.run.vm01.stdout:rabbitmq_rabbitmq-server-source 390 B/s | 819 B 00:02 2026-03-20T18:02:07.006 INFO:teuthology.orchestra.run.vm01.stdout:rabbitmq_rabbitmq-server-source 7.6 kB/s | 3.9 kB 00:00 2026-03-20T18:02:07.037 INFO:teuthology.orchestra.run.vm01.stderr:Importing GPG key 0x4D206F89: 2026-03-20T18:02:07.037 INFO:teuthology.orchestra.run.vm01.stderr: Userid : "https://packagecloud.io/rabbitmq/rabbitmq-server (https://packagecloud.io/docs#gpg_signing) " 2026-03-20T18:02:07.037 INFO:teuthology.orchestra.run.vm01.stderr: Fingerprint: 8C69 5B02 19AF DEB0 4A05 8ED8 F4E7 8920 4D20 6F89 2026-03-20T18:02:07.037 INFO:teuthology.orchestra.run.vm01.stderr: From : https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey 2026-03-20T18:02:10.100 INFO:teuthology.orchestra.run.vm01.stdout:rabbitmq_rabbitmq-server-source 97 B/s | 296 B 00:03 2026-03-20T18:02:10.430 INFO:teuthology.orchestra.run.vm01.stdout:Package yum-utils-4.3.0-26.el9.noarch is already installed. 2026-03-20T18:02:10.457 INFO:teuthology.orchestra.run.vm01.stdout:Dependencies resolved. 2026-03-20T18:02:10.457 INFO:teuthology.orchestra.run.vm01.stdout:Nothing to do. 2026-03-20T18:02:10.457 INFO:teuthology.orchestra.run.vm01.stdout:Complete! 2026-03-20T18:02:10.558 INFO:teuthology.orchestra.run.vm01.stdout:Generating yum cache for rabbitmq_rabbitmq-server... 2026-03-20T18:02:13.030 INFO:teuthology.orchestra.run.vm01.stderr:Importing GPG key 0x4D206F89: 2026-03-20T18:02:13.031 INFO:teuthology.orchestra.run.vm01.stderr: Userid : "https://packagecloud.io/rabbitmq/rabbitmq-server (https://packagecloud.io/docs#gpg_signing) " 2026-03-20T18:02:13.031 INFO:teuthology.orchestra.run.vm01.stderr: Fingerprint: 8C69 5B02 19AF DEB0 4A05 8ED8 F4E7 8920 4D20 6F89 2026-03-20T18:02:13.031 INFO:teuthology.orchestra.run.vm01.stderr: From : https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey 2026-03-20T18:02:15.653 INFO:teuthology.orchestra.run.vm01.stdout:Generating yum cache for rabbitmq_rabbitmq-server-source... 2026-03-20T18:02:17.384 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:02:17.384 INFO:teuthology.orchestra.run.vm01.stdout:The repository is setup! You can now install packages. 2026-03-20T18:02:17.386 DEBUG:teuthology.orchestra.run.vm01:> sudo dnf -y install rabbitmq-server 2026-03-20T18:02:18.067 INFO:teuthology.orchestra.run.vm01.stdout:Dependencies resolved. 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout: Package Arch Version Repository Size 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout:Installing: 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout: rabbitmq-server noarch 3.13.7-1.el8 rabbitmq_rabbitmq-server 18 M 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout:Transaction Summary 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout:Install 1 Package 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout:Total download size: 18 M 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout:Installed size: 24 M 2026-03-20T18:02:18.068 INFO:teuthology.orchestra.run.vm01.stdout:Downloading Packages: 2026-03-20T18:02:18.852 INFO:teuthology.orchestra.run.vm01.stdout:rabbitmq-server-3.13.7-1.el8.noarch.rpm 23 MB/s | 18 MB 00:00 2026-03-20T18:02:18.854 INFO:teuthology.orchestra.run.vm01.stdout:-------------------------------------------------------------------------------- 2026-03-20T18:02:18.854 INFO:teuthology.orchestra.run.vm01.stdout:Total 23 MB/s | 18 MB 00:00 2026-03-20T18:02:18.854 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction check 2026-03-20T18:02:18.859 INFO:teuthology.orchestra.run.vm01.stdout:Transaction check succeeded. 2026-03-20T18:02:18.859 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction test 2026-03-20T18:02:18.939 INFO:teuthology.orchestra.run.vm01.stdout:Transaction test succeeded. 2026-03-20T18:02:18.939 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction 2026-03-20T18:02:19.035 INFO:teuthology.orchestra.run.vm01.stdout: Preparing : 1/1 2026-03-20T18:02:19.635 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: rabbitmq-server-3.13.7-1.el8.noarch 1/1 2026-03-20T18:02:19.642 INFO:teuthology.orchestra.run.vm01.stdout: Installing : rabbitmq-server-3.13.7-1.el8.noarch 1/1 2026-03-20T18:02:19.953 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: rabbitmq-server-3.13.7-1.el8.noarch 1/1 2026-03-20T18:02:19.954 INFO:teuthology.orchestra.run.vm01.stdout:/usr/lib/tmpfiles.d/rabbitmq-server.conf:1: Line references path below legacy directory /var/run/, updating /var/run/rabbitmq → /run/rabbitmq; please update the tmpfiles.d/ drop-in file accordingly. 2026-03-20T18:02:19.954 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:02:20.282 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : rabbitmq-server-3.13.7-1.el8.noarch 1/1 2026-03-20T18:02:20.282 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:02:20.282 INFO:teuthology.orchestra.run.vm01.stdout:Installed: 2026-03-20T18:02:20.282 INFO:teuthology.orchestra.run.vm01.stdout: rabbitmq-server-3.13.7-1.el8.noarch 2026-03-20T18:02:20.282 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:02:20.282 INFO:teuthology.orchestra.run.vm01.stdout:Complete! 2026-03-20T18:02:20.317 INFO:tasks.rabbitmq:Bringing up Daemon and RabbitMQ service... 2026-03-20T18:02:20.317 DEBUG:teuthology.orchestra.run.vm01:> echo 'loopback_users.guest = false' | sudo tee -a /etc/rabbitmq/rabbitmq.conf 2026-03-20T18:02:20.345 INFO:teuthology.orchestra.run.vm01.stdout:loopback_users.guest = false 2026-03-20T18:02:20.346 DEBUG:teuthology.orchestra.run.vm01:> sudo systemctl enable rabbitmq-server 2026-03-20T18:02:20.415 INFO:teuthology.orchestra.run.vm01.stderr:Created symlink /etc/systemd/system/multi-user.target.wants/rabbitmq-server.service → /usr/lib/systemd/system/rabbitmq-server.service. 2026-03-20T18:02:20.531 DEBUG:teuthology.orchestra.run.vm01:> sudo systemctl start rabbitmq-server 2026-03-20T18:02:22.541 DEBUG:teuthology.orchestra.run.vm01:> sudo systemctl status rabbitmq-server 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout:● rabbitmq-server.service - RabbitMQ broker 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; preset: disabled) 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: Active: active (running) since Fri 2026-03-20 18:02:22 UTC; 33ms ago 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: Main PID: 65384 (beam.smp) 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: Tasks: 32 (limit: 60479) 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: Memory: 145.4M (peak: 146.5M) 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: CPU: 2.057s 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: CGroup: /system.slice/rabbitmq-server.service 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: ├─65384 /usr/lib64/erlang/erts-14.2.5.4/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -pc unicode -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio none -- -root /usr/lib64/erlang -bindir /usr/lib64/erlang/erts-14.2.5.4/bin -progname erl -- -home /var/lib/rabbitmq -- -pa "" -noshell -noinput -s rabbit boot -boot start_sasl -syslog logger "[]" -syslog syslog_error_logger false -kernel prevent_overlapping_partitions false 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: ├─65397 erl_child_setup 32768 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: ├─65418 sh -s disksup 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: ├─65420 /usr/lib64/erlang/lib/os_mon-2.9.1/priv/bin/memsup 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: ├─65421 /usr/lib64/erlang/lib/os_mon-2.9.1/priv/bin/cpu_sup 2026-03-20T18:02:22.573 INFO:teuthology.orchestra.run.vm01.stdout: ├─65422 /usr/lib64/erlang/erts-14.2.5.4/bin/inet_gethost 4 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout: ├─65423 /usr/lib64/erlang/erts-14.2.5.4/bin/inet_gethost 4 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout: ├─65432 /usr/lib64/erlang/erts-14.2.5.4/bin/epmd -daemon 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout: └─65459 /bin/sh -s rabbit_disk_monitor 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout:Mar 20 18:02:22 vm01.local rabbitmq-server[65384]: Doc guides: https://www.rabbitmq.com/docs 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout:Mar 20 18:02:22 vm01.local rabbitmq-server[65384]: Support: https://www.rabbitmq.com/docs/contact 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout:Mar 20 18:02:22 vm01.local rabbitmq-server[65384]: Tutorials: https://www.rabbitmq.com/tutorials 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout:Mar 20 18:02:22 vm01.local rabbitmq-server[65384]: Monitoring: https://www.rabbitmq.com/docs/monitoring 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout:Mar 20 18:02:22 vm01.local rabbitmq-server[65384]: Upgrading: https://www.rabbitmq.com/docs/upgrade 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout:Mar 20 18:02:22 vm01.local rabbitmq-server[65384]: Logs: /var/log/rabbitmq/rabbit@vm01.log 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout:Mar 20 18:02:22 vm01.local rabbitmq-server[65384]: 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout:Mar 20 18:02:22 vm01.local rabbitmq-server[65384]: Config file(s): /etc/rabbitmq/rabbitmq.conf 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout:Mar 20 18:02:22 vm01.local rabbitmq-server[65384]: Starting broker... completed with 0 plugins. 2026-03-20T18:02:22.574 INFO:teuthology.orchestra.run.vm01.stdout:Mar 20 18:02:22 vm01.local systemd[1]: Started RabbitMQ broker. 2026-03-20T18:02:22.575 INFO:teuthology.run_tasks:Running task notification-tests... 2026-03-20T18:02:22.579 DEBUG:tasks.notification_tests:Notifications config is {'client.0': {'extra_attr': ['amqp_test'], 'rgw_server': 'client.0'}} 2026-03-20T18:02:22.579 INFO:tasks.notification_tests:Downloading bucket-notifications-tests... 2026-03-20T18:02:22.579 INFO:tasks.notification_tests:Using branch tt-tentacle from https://github.com/kshtsk/ceph.git for bucket notifications tests 2026-03-20T18:02:22.579 DEBUG:teuthology.orchestra.run.vm01:> git clone -b tt-tentacle https://github.com/kshtsk/ceph.git /home/ubuntu/cephtest/ceph 2026-03-20T18:02:22.632 INFO:teuthology.orchestra.run.vm01.stderr:Cloning into '/home/ubuntu/cephtest/ceph'... 2026-03-20T18:03:25.002 INFO:teuthology.orchestra.run.vm01.stderr:Updating files: 96% (13219/13634) Updating files: 97% (13225/13634) Updating files: 98% (13362/13634) Updating files: 99% (13498/13634) Updating files: 100% (13634/13634) Updating files: 100% (13634/13634), done. 2026-03-20T18:03:25.024 INFO:tasks.notification_tests:Pre-processing... 2026-03-20T18:03:25.025 DEBUG:teuthology.orchestra.run.vm01:> mkdir -p /home/ubuntu/.aws/models/s3/2006-03-01/ 2026-03-20T18:03:25.081 DEBUG:teuthology.orchestra.run.vm01:> cd /home/ubuntu/.aws/models/s3/2006-03-01/ && cp /home/ubuntu/cephtest/ceph/examples/rgw/boto3/service-2.sdk-extras.json service-2.sdk-extras.json 2026-03-20T18:03:25.137 INFO:tasks.notification_tests:Creating rgw user... 2026-03-20T18:03:25.137 DEBUG:tasks.notification_tests:Creating user foo.client.0 on client.0 2026-03-20T18:03:25.138 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key QHRUKFLLXTHPEBHYWCSR --secret P6uO8Klf1nBAwB0wsOHf6n6vY5s99vrOu3n0QFuHUhtmAAjIqYua0A== --cluster ceph 2026-03-20T18:03:25.215 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:03:25.215 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:03:25.219 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.220+0000 7febc4b30900 -1 asok(0x55599868c640) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.0.65580.asok': (13) Permission denied 2026-03-20T18:03:25.236 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.237+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.238+0000 7febc4b30900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:03:25.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.238+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.238 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.239+0000 7febc4b30900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:03:25.238 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.239+0000 7febc4b30900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:03:25.238 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.239+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.238 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.239+0000 7febc4b30900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:03:25.238 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.239+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.239 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.239+0000 7febc4b30900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:03:25.239 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.239+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.239 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.240+0000 7febc4b30900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:03:25.240 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.240+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.240 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.241+0000 7febc4b30900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:03:25.240 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.241+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.240 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.241+0000 7febc4b30900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:03:25.240 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.241+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.241 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.241+0000 7febc4b30900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:03:25.241 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.241+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.242+0000 7febc4b30900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.242+0000 7febc4b30900 20 zone MyZone found 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.242+0000 7febc4b30900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.242+0000 7febc4b30900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.242+0000 7febc4b30900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.242+0000 7febc4b30900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.242+0000 7febc4b30900 20 period zonegroup name MyZoneGroup 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.242+0000 7febc4b30900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.242+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.243+0000 7febc4b30900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.243+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.243+0000 7febc4b30900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.243+0000 7febc4b30900 20 started sync module instance, tier type = 2026-03-20T18:03:25.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.243+0000 7febc4b30900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:03:25.246 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.247+0000 7febc4b30900 20 add_watcher() i=0 2026-03-20T18:03:25.247 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.248+0000 7febc4b30900 20 add_watcher() i=2 2026-03-20T18:03:25.247 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.248+0000 7febc4b30900 20 add_watcher() i=5 2026-03-20T18:03:25.247 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.248+0000 7febc4b30900 20 add_watcher() i=6 2026-03-20T18:03:25.247 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.248+0000 7febc4b30900 20 add_watcher() i=4 2026-03-20T18:03:25.247 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.248+0000 7febc4b30900 20 add_watcher() i=1 2026-03-20T18:03:25.248 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.249+0000 7febc4b30900 20 add_watcher() i=3 2026-03-20T18:03:25.248 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.249+0000 7febc4b30900 20 add_watcher() i=7 2026-03-20T18:03:25.248 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.249+0000 7febc4b30900 2 all 8 watchers are set, enabling cache 2026-03-20T18:03:25.250 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.251+0000 7febc4b30900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:03:25.250 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.251+0000 7febc4b30900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:03:25.250 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.251+0000 7febc4b30900 5 note: GC not initialized 2026-03-20T18:03:25.250 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.251+0000 7feb72fe5640 20 reqs_thread_entry: start 2026-03-20T18:03:25.298 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 20 init_complete bucket index max shards: 11 2026-03-20T18:03:25.298 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 20 Filter name: none 2026-03-20T18:03:25.298 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7feb70fe1640 20 reqs_thread_entry: start 2026-03-20T18:03:25.298 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 10 cache get: name=MyZone.rgw.meta+users.uid+foo.client.0 : miss 2026-03-20T18:03:25.298 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.298 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:03:25.298 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 10 cache put: name=MyZone.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2026-03-20T18:03:25.298 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 10 adding MyZone.rgw.meta+users.uid+foo.client.0 to cache LRU end 2026-03-20T18:03:25.298 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 10 cache get: name=MyZone.rgw.meta+users.keys+QHRUKFLLXTHPEBHYWCSR : miss 2026-03-20T18:03:25.298 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 20 rados->read ofs=0 len=0 2026-03-20T18:03:25.299 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:03:25.299 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 10 cache put: name=MyZone.rgw.meta+users.keys+QHRUKFLLXTHPEBHYWCSR info.flags=0x0 2026-03-20T18:03:25.299 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 10 adding MyZone.rgw.meta+users.keys+QHRUKFLLXTHPEBHYWCSR to cache LRU end 2026-03-20T18:03:25.299 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 10 cache get: name=MyZone.rgw.meta+users.keys+QHRUKFLLXTHPEBHYWCSR : hit (negative entry) 2026-03-20T18:03:25.299 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.299+0000 7febc4b30900 10 cache get: name=MyZone.rgw.meta+users.keys+QHRUKFLLXTHPEBHYWCSR : hit (negative entry) 2026-03-20T18:03:25.300 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.300+0000 7febc4b30900 10 cache put: name=MyZone.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2026-03-20T18:03:25.300 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.300+0000 7febc4b30900 10 moving MyZone.rgw.meta+users.uid+foo.client.0 to cache LRU end 2026-03-20T18:03:25.300 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.301+0000 7febc4b30900 10 distributing notification oid=MyZone.rgw.control:notify.0 cni=[op: 0, obj: MyZone.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2026-03-20T18:03:25.300 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.301+0000 7feb8ffff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659136 cookie 93843303910800 notifier 4595 bl.length()=627 2026-03-20T18:03:25.300 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.301+0000 7feb8ffff640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2026-03-20T18:03:25.300 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.301+0000 7feb8ffff640 10 rgw watcher librados: moving MyZone.rgw.meta+users.uid+foo.client.0 to cache LRU end 2026-03-20T18:03:25.309 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.310+0000 7febc4b30900 10 cache put: name=MyZone.rgw.meta+users.keys+QHRUKFLLXTHPEBHYWCSR info.flags=0x7 2026-03-20T18:03:25.309 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.310+0000 7febc4b30900 10 moving MyZone.rgw.meta+users.keys+QHRUKFLLXTHPEBHYWCSR to cache LRU end 2026-03-20T18:03:25.309 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.310+0000 7febc4b30900 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 0, obj: MyZone.rgw.meta:users.keys:QHRUKFLLXTHPEBHYWCSR, ofs0, ns] 2026-03-20T18:03:25.309 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.310+0000 7feb8ffff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659136 cookie 93843303754336 notifier 4595 bl.length()=185 2026-03-20T18:03:25.309 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.310+0000 7feb8ffff640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.keys+QHRUKFLLXTHPEBHYWCSR info.flags=0x7 2026-03-20T18:03:25.309 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.310+0000 7feb8ffff640 10 rgw watcher librados: moving MyZone.rgw.meta+users.keys+QHRUKFLLXTHPEBHYWCSR to cache LRU end 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout:{ 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "user_id": "foo.client.0", 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "display_name": "Mr. foo.client.0", 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "email": "", 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "suspended": 0, 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "max_buckets": 1000, 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "subusers": [], 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "keys": [ 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: { 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "user": "foo.client.0", 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "access_key": "QHRUKFLLXTHPEBHYWCSR", 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "secret_key": "P6uO8Klf1nBAwB0wsOHf6n6vY5s99vrOu3n0QFuHUhtmAAjIqYua0A==", 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "active": true, 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "create_date": "2026-03-20T18:03:25.301237Z" 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: } 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: ], 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "swift_keys": [], 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "caps": [], 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "op_mask": "read, write, delete", 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "default_placement": "", 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "default_storage_class": "", 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "placement_tags": [], 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "bucket_quota": { 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "enabled": false, 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "check_on_raw": false, 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "max_size": -1, 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "max_size_kb": 0, 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "max_objects": -1 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: }, 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "user_quota": { 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "enabled": false, 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "check_on_raw": false, 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "max_size": -1, 2026-03-20T18:03:25.311 INFO:teuthology.orchestra.run.vm01.stdout: "max_size_kb": 0, 2026-03-20T18:03:25.312 INFO:teuthology.orchestra.run.vm01.stdout: "max_objects": -1 2026-03-20T18:03:25.312 INFO:teuthology.orchestra.run.vm01.stdout: }, 2026-03-20T18:03:25.312 INFO:teuthology.orchestra.run.vm01.stdout: "temp_url_keys": [], 2026-03-20T18:03:25.312 INFO:teuthology.orchestra.run.vm01.stdout: "type": "rgw", 2026-03-20T18:03:25.312 INFO:teuthology.orchestra.run.vm01.stdout: "mfa_ids": [], 2026-03-20T18:03:25.312 INFO:teuthology.orchestra.run.vm01.stdout: "account_id": "", 2026-03-20T18:03:25.312 INFO:teuthology.orchestra.run.vm01.stdout: "path": "/", 2026-03-20T18:03:25.312 INFO:teuthology.orchestra.run.vm01.stdout: "create_date": "2026-03-20T18:03:25.301217Z", 2026-03-20T18:03:25.312 INFO:teuthology.orchestra.run.vm01.stdout: "tags": [], 2026-03-20T18:03:25.312 INFO:teuthology.orchestra.run.vm01.stdout: "group_ids": [] 2026-03-20T18:03:25.312 INFO:teuthology.orchestra.run.vm01.stdout:} 2026-03-20T18:03:25.312 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:25.316 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.317+0000 7febc4b30900 20 remove_watcher() i=0 2026-03-20T18:03:25.316 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.317+0000 7febc4b30900 2 removed watcher, disabling cache 2026-03-20T18:03:25.316 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.317+0000 7febc4b30900 20 remove_watcher() i=7 2026-03-20T18:03:25.317 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.317+0000 7febc4b30900 20 remove_watcher() i=2 2026-03-20T18:03:25.317 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.317+0000 7febc4b30900 20 remove_watcher() i=3 2026-03-20T18:03:25.317 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.317+0000 7febc4b30900 20 remove_watcher() i=4 2026-03-20T18:03:25.317 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.317+0000 7febc4b30900 20 remove_watcher() i=1 2026-03-20T18:03:25.317 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.317+0000 7febc4b30900 20 remove_watcher() i=5 2026-03-20T18:03:25.317 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:25.317+0000 7febc4b30900 20 remove_watcher() i=6 2026-03-20T18:03:25.324 INFO:tasks.notification_tests:Configuring bucket-notifications-tests... 2026-03-20T18:03:25.324 DEBUG:teuthology.orchestra.run.vm01:> set -ex 2026-03-20T18:03:25.324 DEBUG:teuthology.orchestra.run.vm01:> dd of=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf 2026-03-20T18:03:25.380 DEBUG:teuthology.orchestra.run.vm01:> cd /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification && ./bootstrap 2026-03-20T18:03:26.237 INFO:teuthology.orchestra.run.vm01.stderr:./bootstrap: missing required RPM packages. Installing via sudo. 2026-03-20T18:03:26.817 INFO:teuthology.orchestra.run.vm01.stdout:Last metadata expiration check: 0:01:09 ago on Fri 20 Mar 2026 06:02:17 PM UTC. 2026-03-20T18:03:26.927 INFO:teuthology.orchestra.run.vm01.stdout:Dependencies resolved. 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: Package Arch Version Repository Size 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout:Installing: 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: libevent-devel x86_64 2.1.12-8.el9 appstream 99 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: libxml2-devel x86_64 2.9.13-14.el9 appstream 899 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: libxslt-devel x86_64 1.1.34-12.el9 appstream 319 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: python3-pika noarch 1.2.1-1.el9 epel 244 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout:Installing dependencies: 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: cmake-filesystem x86_64 3.31.8-3.el9 appstream 19 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: libgpg-error-devel x86_64 1.42-5.el9 appstream 66 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: python3-Automat noarch 22.10.0-2.el9 epel 59 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: python3-constantly noarch 15.1.0-25.el9 epel 20 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: python3-hyperlink noarch 21.0.0-10.el9 epel 115 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: python3-incremental noarch 21.3.0-5.el9 epel 34 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: python3-service-identity noarch 21.1.0-5.el9 epel 27 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: python3-zope-interface x86_64 5.4.0-5.el9.1 epel 168 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: xz-devel x86_64 5.2.5-8.el9 appstream 55 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout:Installing weak dependencies: 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: python3-tornado x86_64 6.4.2-3.el9 appstream 720 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: python3-twisted noarch 22.10.0-3.el9 epel 5.1 M 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: python3-twisted+tls noarch 22.10.0-3.el9 epel 9.4 k 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout:Transaction Summary 2026-03-20T18:03:26.928 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:03:26.929 INFO:teuthology.orchestra.run.vm01.stdout:Install 17 Packages 2026-03-20T18:03:26.929 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:26.929 INFO:teuthology.orchestra.run.vm01.stdout:Total download size: 7.9 M 2026-03-20T18:03:26.929 INFO:teuthology.orchestra.run.vm01.stdout:Installed size: 47 M 2026-03-20T18:03:26.929 INFO:teuthology.orchestra.run.vm01.stdout:Downloading Packages: 2026-03-20T18:03:27.440 INFO:teuthology.orchestra.run.vm01.stdout:(1/17): cmake-filesystem-3.31.8-3.el9.x86_64.rp 82 kB/s | 19 kB 00:00 2026-03-20T18:03:27.487 INFO:teuthology.orchestra.run.vm01.stdout:(2/17): libgpg-error-devel-1.42-5.el9.x86_64.rp 238 kB/s | 66 kB 00:00 2026-03-20T18:03:27.513 INFO:teuthology.orchestra.run.vm01.stdout:(3/17): libevent-devel-2.1.12-8.el9.x86_64.rpm 325 kB/s | 99 kB 00:00 2026-03-20T18:03:27.751 INFO:teuthology.orchestra.run.vm01.stdout:(4/17): libxslt-devel-1.1.34-12.el9.x86_64.rpm 1.2 MB/s | 319 kB 00:00 2026-03-20T18:03:27.860 INFO:teuthology.orchestra.run.vm01.stdout:(5/17): python3-tornado-6.4.2-3.el9.x86_64.rpm 2.0 MB/s | 720 kB 00:00 2026-03-20T18:03:27.868 INFO:teuthology.orchestra.run.vm01.stdout:(6/17): xz-devel-5.2.5-8.el9.x86_64.rpm 474 kB/s | 55 kB 00:00 2026-03-20T18:03:27.893 INFO:teuthology.orchestra.run.vm01.stdout:(7/17): libxml2-devel-2.9.13-14.el9.x86_64.rpm 1.9 MB/s | 899 kB 00:00 2026-03-20T18:03:27.907 INFO:teuthology.orchestra.run.vm01.stdout:(8/17): python3-Automat-22.10.0-2.el9.noarch.rp 1.5 MB/s | 59 kB 00:00 2026-03-20T18:03:27.920 INFO:teuthology.orchestra.run.vm01.stdout:(9/17): zlib-devel-1.2.11-41.el9.x86_64.rpm 754 kB/s | 45 kB 00:00 2026-03-20T18:03:27.921 INFO:teuthology.orchestra.run.vm01.stdout:(10/17): python3-constantly-15.1.0-25.el9.noarc 700 kB/s | 20 kB 00:00 2026-03-20T18:03:27.938 INFO:teuthology.orchestra.run.vm01.stdout:(11/17): python3-incremental-21.3.0-5.el9.noarc 2.0 MB/s | 34 kB 00:00 2026-03-20T18:03:27.943 INFO:teuthology.orchestra.run.vm01.stdout:(12/17): python3-pika-1.2.1-1.el9.noarch.rpm 11 MB/s | 244 kB 00:00 2026-03-20T18:03:27.947 INFO:teuthology.orchestra.run.vm01.stdout:(13/17): python3-hyperlink-21.0.0-10.el9.noarch 2.9 MB/s | 115 kB 00:00 2026-03-20T18:03:27.954 INFO:teuthology.orchestra.run.vm01.stdout:(14/17): python3-service-identity-21.1.0-5.el9. 1.7 MB/s | 27 kB 00:00 2026-03-20T18:03:27.960 INFO:teuthology.orchestra.run.vm01.stdout:(15/17): python3-twisted+tls-22.10.0-3.el9.noar 563 kB/s | 9.4 kB 00:00 2026-03-20T18:03:28.116 INFO:teuthology.orchestra.run.vm01.stdout:(16/17): python3-zope-interface-5.4.0-5.el9.1.x 1.0 MB/s | 168 kB 00:00 2026-03-20T18:03:28.218 INFO:teuthology.orchestra.run.vm01.stdout:(17/17): python3-twisted-22.10.0-3.el9.noarch.r 19 MB/s | 5.1 MB 00:00 2026-03-20T18:03:28.218 INFO:teuthology.orchestra.run.vm01.stdout:-------------------------------------------------------------------------------- 2026-03-20T18:03:28.218 INFO:teuthology.orchestra.run.vm01.stdout:Total 6.1 MB/s | 7.9 MB 00:01 2026-03-20T18:03:28.292 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction check 2026-03-20T18:03:28.302 INFO:teuthology.orchestra.run.vm01.stdout:Transaction check succeeded. 2026-03-20T18:03:28.302 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction test 2026-03-20T18:03:28.406 INFO:teuthology.orchestra.run.vm01.stdout:Transaction test succeeded. 2026-03-20T18:03:28.406 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction 2026-03-20T18:03:28.549 INFO:teuthology.orchestra.run.vm01.stdout: Preparing : 1/1 2026-03-20T18:03:28.558 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-zope-interface-5.4.0-5.el9.1.x86_64 1/17 2026-03-20T18:03:28.566 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-service-identity-21.1.0-5.el9.noarch 2/17 2026-03-20T18:03:28.580 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-incremental-21.3.0-5.el9.noarch 3/17 2026-03-20T18:03:28.587 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-hyperlink-21.0.0-10.el9.noarch 4/17 2026-03-20T18:03:28.597 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-constantly-15.1.0-25.el9.noarch 5/17 2026-03-20T18:03:28.599 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-Automat-22.10.0-2.el9.noarch 6/17 2026-03-20T18:03:29.259 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-twisted+tls-22.10.0-3.el9.noarch 7/17 2026-03-20T18:03:29.278 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-twisted-22.10.0-3.el9.noarch 8/17 2026-03-20T18:03:29.285 INFO:teuthology.orchestra.run.vm01.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 9/17 2026-03-20T18:03:29.353 INFO:teuthology.orchestra.run.vm01.stdout: Installing : xz-devel-5.2.5-8.el9.x86_64 10/17 2026-03-20T18:03:29.361 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-tornado-6.4.2-3.el9.x86_64 11/17 2026-03-20T18:03:29.368 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libgpg-error-devel-1.42-5.el9.x86_64 12/17 2026-03-20T18:03:29.470 INFO:teuthology.orchestra.run.vm01.stdout: Installing : cmake-filesystem-3.31.8-3.el9.x86_64 13/17 2026-03-20T18:03:29.516 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libxml2-devel-2.9.13-14.el9.x86_64 14/17 2026-03-20T18:03:29.546 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libxslt-devel-1.1.34-12.el9.x86_64 15/17 2026-03-20T18:03:29.558 INFO:teuthology.orchestra.run.vm01.stdout: Installing : python3-pika-1.2.1-1.el9.noarch 16/17 2026-03-20T18:03:29.576 INFO:teuthology.orchestra.run.vm01.stdout: Installing : libevent-devel-2.1.12-8.el9.x86_64 17/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: libevent-devel-2.1.12-8.el9.x86_64 17/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : cmake-filesystem-3.31.8-3.el9.x86_64 1/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libevent-devel-2.1.12-8.el9.x86_64 2/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libgpg-error-devel-1.42-5.el9.x86_64 3/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libxml2-devel-2.9.13-14.el9.x86_64 4/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : libxslt-devel-1.1.34-12.el9.x86_64 5/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-tornado-6.4.2-3.el9.x86_64 6/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : xz-devel-5.2.5-8.el9.x86_64 7/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 8/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-Automat-22.10.0-2.el9.noarch 9/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-constantly-15.1.0-25.el9.noarch 10/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-hyperlink-21.0.0-10.el9.noarch 11/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-incremental-21.3.0-5.el9.noarch 12/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-pika-1.2.1-1.el9.noarch 13/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-service-identity-21.1.0-5.el9.noarch 14/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-twisted+tls-22.10.0-3.el9.noarch 15/17 2026-03-20T18:03:30.205 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-twisted-22.10.0-3.el9.noarch 16/17 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : python3-zope-interface-5.4.0-5.el9.1.x86_64 17/17 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout:Installed: 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: cmake-filesystem-3.31.8-3.el9.x86_64 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: libevent-devel-2.1.12-8.el9.x86_64 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: libgpg-error-devel-1.42-5.el9.x86_64 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: libxml2-devel-2.9.13-14.el9.x86_64 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: libxslt-devel-1.1.34-12.el9.x86_64 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: python3-Automat-22.10.0-2.el9.noarch 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: python3-constantly-15.1.0-25.el9.noarch 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: python3-hyperlink-21.0.0-10.el9.noarch 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: python3-incremental-21.3.0-5.el9.noarch 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: python3-pika-1.2.1-1.el9.noarch 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: python3-service-identity-21.1.0-5.el9.noarch 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: python3-tornado-6.4.2-3.el9.x86_64 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: python3-twisted-22.10.0-3.el9.noarch 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: python3-twisted+tls-22.10.0-3.el9.noarch 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: python3-zope-interface-5.4.0-5.el9.1.x86_64 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: xz-devel-5.2.5-8.el9.x86_64 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: zlib-devel-1.2.11-41.el9.x86_64 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:30.287 INFO:teuthology.orchestra.run.vm01.stdout:Complete! 2026-03-20T18:03:31.695 INFO:teuthology.orchestra.run.vm01.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.9/site-packages (21.3.1) 2026-03-20T18:03:31.804 INFO:teuthology.orchestra.run.vm01.stdout:Collecting pip 2026-03-20T18:03:31.833 INFO:teuthology.orchestra.run.vm01.stdout: Downloading pip-26.0.1-py3-none-any.whl (1.8 MB) 2026-03-20T18:03:32.033 INFO:teuthology.orchestra.run.vm01.stdout:Installing collected packages: pip 2026-03-20T18:03:32.033 INFO:teuthology.orchestra.run.vm01.stdout: Attempting uninstall: pip 2026-03-20T18:03:32.033 INFO:teuthology.orchestra.run.vm01.stdout: Found existing installation: pip 21.3.1 2026-03-20T18:03:32.161 INFO:teuthology.orchestra.run.vm01.stdout: Uninstalling pip-21.3.1: 2026-03-20T18:03:32.164 INFO:teuthology.orchestra.run.vm01.stdout: Successfully uninstalled pip-21.3.1 2026-03-20T18:03:32.603 INFO:teuthology.orchestra.run.vm01.stdout:Successfully installed pip-26.0.1 2026-03-20T18:03:33.025 INFO:teuthology.orchestra.run.vm01.stdout:Collecting pynose (from -r requirements.txt (line 1)) 2026-03-20T18:03:33.056 INFO:teuthology.orchestra.run.vm01.stdout: Downloading pynose-1.5.5-py3-none-any.whl.metadata (19 kB) 2026-03-20T18:03:33.078 INFO:teuthology.orchestra.run.vm01.stdout:Collecting boto>=2.6.0 (from -r requirements.txt (line 2)) 2026-03-20T18:03:33.087 INFO:teuthology.orchestra.run.vm01.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl.metadata (7.3 kB) 2026-03-20T18:03:33.307 INFO:teuthology.orchestra.run.vm01.stdout:Collecting boto3>=1.0.0 (from -r requirements.txt (line 3)) 2026-03-20T18:03:33.316 INFO:teuthology.orchestra.run.vm01.stdout: Downloading boto3-1.42.72-py3-none-any.whl.metadata (6.7 kB) 2026-03-20T18:03:33.334 INFO:teuthology.orchestra.run.vm01.stdout:Collecting configparser>=5.0.0 (from -r requirements.txt (line 4)) 2026-03-20T18:03:33.343 INFO:teuthology.orchestra.run.vm01.stdout: Downloading configparser-7.2.0-py3-none-any.whl.metadata (5.5 kB) 2026-03-20T18:03:33.362 INFO:teuthology.orchestra.run.vm01.stdout:Collecting kafka-python>=2.0.0 (from -r requirements.txt (line 5)) 2026-03-20T18:03:33.371 INFO:teuthology.orchestra.run.vm01.stdout: Downloading kafka_python-2.3.0-py2.py3-none-any.whl.metadata (10.0 kB) 2026-03-20T18:03:33.374 INFO:teuthology.orchestra.run.vm01.stdout:Requirement already satisfied: pika in /usr/lib/python3.9/site-packages (from -r requirements.txt (line 6)) (1.2.1) 2026-03-20T18:03:33.387 INFO:teuthology.orchestra.run.vm01.stdout:Collecting pika (from -r requirements.txt (line 6)) 2026-03-20T18:03:33.396 INFO:teuthology.orchestra.run.vm01.stdout: Downloading pika-1.3.2-py3-none-any.whl.metadata (13 kB) 2026-03-20T18:03:33.414 INFO:teuthology.orchestra.run.vm01.stdout:Collecting cloudevents (from -r requirements.txt (line 7)) 2026-03-20T18:03:33.425 INFO:teuthology.orchestra.run.vm01.stdout: Downloading cloudevents-1.12.1-py3-none-any.whl.metadata (7.2 kB) 2026-03-20T18:03:33.427 INFO:teuthology.orchestra.run.vm01.stdout:Requirement already satisfied: xmltodict in /usr/lib/python3.9/site-packages (from -r requirements.txt (line 8)) (0.12.0) 2026-03-20T18:03:33.440 INFO:teuthology.orchestra.run.vm01.stdout:Collecting xmltodict (from -r requirements.txt (line 8)) 2026-03-20T18:03:33.448 INFO:teuthology.orchestra.run.vm01.stdout: Downloading xmltodict-1.0.4-py3-none-any.whl.metadata (14 kB) 2026-03-20T18:03:33.627 INFO:teuthology.orchestra.run.vm01.stdout:Collecting botocore<1.43.0,>=1.42.72 (from boto3>=1.0.0->-r requirements.txt (line 3)) 2026-03-20T18:03:33.635 INFO:teuthology.orchestra.run.vm01.stdout: Downloading botocore-1.42.72-py3-none-any.whl.metadata (5.9 kB) 2026-03-20T18:03:33.638 INFO:teuthology.orchestra.run.vm01.stdout:Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3>=1.0.0->-r requirements.txt (line 3)) (1.0.1) 2026-03-20T18:03:33.653 INFO:teuthology.orchestra.run.vm01.stdout:Collecting s3transfer<0.17.0,>=0.16.0 (from boto3>=1.0.0->-r requirements.txt (line 3)) 2026-03-20T18:03:33.661 INFO:teuthology.orchestra.run.vm01.stdout: Downloading s3transfer-0.16.0-py3-none-any.whl.metadata (1.7 kB) 2026-03-20T18:03:33.665 INFO:teuthology.orchestra.run.vm01.stdout:Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.43.0,>=1.42.72->boto3>=1.0.0->-r requirements.txt (line 3)) (2.9.0.post0) 2026-03-20T18:03:33.665 INFO:teuthology.orchestra.run.vm01.stdout:Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.43.0,>=1.42.72->boto3>=1.0.0->-r requirements.txt (line 3)) (1.26.5) 2026-03-20T18:03:33.666 INFO:teuthology.orchestra.run.vm01.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.43.0,>=1.42.72->boto3>=1.0.0->-r requirements.txt (line 3)) (1.15.0) 2026-03-20T18:03:33.704 INFO:teuthology.orchestra.run.vm01.stdout:Collecting deprecation<3.0,>=2.0 (from cloudevents->-r requirements.txt (line 7)) 2026-03-20T18:03:33.712 INFO:teuthology.orchestra.run.vm01.stdout: Downloading deprecation-2.1.0-py2.py3-none-any.whl.metadata (4.6 kB) 2026-03-20T18:03:33.715 INFO:teuthology.orchestra.run.vm01.stdout:Requirement already satisfied: packaging in /usr/lib/python3.9/site-packages (from deprecation<3.0,>=2.0->cloudevents->-r requirements.txt (line 7)) (20.9) 2026-03-20T18:03:33.716 INFO:teuthology.orchestra.run.vm01.stdout:Requirement already satisfied: pyparsing>=2.0.2 in /usr/lib/python3.9/site-packages (from packaging->deprecation<3.0,>=2.0->cloudevents->-r requirements.txt (line 7)) (2.4.7) 2026-03-20T18:03:33.726 INFO:teuthology.orchestra.run.vm01.stdout:Downloading pynose-1.5.5-py3-none-any.whl (130 kB) 2026-03-20T18:03:33.746 INFO:teuthology.orchestra.run.vm01.stdout:Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2026-03-20T18:03:33.772 INFO:teuthology.orchestra.run.vm01.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 62.1 MB/s 0:00:00 2026-03-20T18:03:33.781 INFO:teuthology.orchestra.run.vm01.stdout:Downloading boto3-1.42.72-py3-none-any.whl (140 kB) 2026-03-20T18:03:33.792 INFO:teuthology.orchestra.run.vm01.stdout:Downloading botocore-1.42.72-py3-none-any.whl (14.7 MB) 2026-03-20T18:03:33.961 INFO:teuthology.orchestra.run.vm01.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.7/14.7 MB 86.7 MB/s 0:00:00 2026-03-20T18:03:33.970 INFO:teuthology.orchestra.run.vm01.stdout:Downloading s3transfer-0.16.0-py3-none-any.whl (86 kB) 2026-03-20T18:03:33.981 INFO:teuthology.orchestra.run.vm01.stdout:Downloading configparser-7.2.0-py3-none-any.whl (17 kB) 2026-03-20T18:03:33.992 INFO:teuthology.orchestra.run.vm01.stdout:Downloading kafka_python-2.3.0-py2.py3-none-any.whl (326 kB) 2026-03-20T18:03:34.006 INFO:teuthology.orchestra.run.vm01.stdout:Downloading pika-1.3.2-py3-none-any.whl (155 kB) 2026-03-20T18:03:34.019 INFO:teuthology.orchestra.run.vm01.stdout:Downloading cloudevents-1.12.1-py3-none-any.whl (55 kB) 2026-03-20T18:03:34.030 INFO:teuthology.orchestra.run.vm01.stdout:Downloading deprecation-2.1.0-py2.py3-none-any.whl (11 kB) 2026-03-20T18:03:34.040 INFO:teuthology.orchestra.run.vm01.stdout:Downloading xmltodict-1.0.4-py3-none-any.whl (13 kB) 2026-03-20T18:03:34.113 INFO:teuthology.orchestra.run.vm01.stdout:Installing collected packages: kafka-python, boto, xmltodict, pynose, pika, configparser, deprecation, botocore, s3transfer, cloudevents, boto3 2026-03-20T18:03:34.501 INFO:teuthology.orchestra.run.vm01.stdout: Attempting uninstall: xmltodict 2026-03-20T18:03:34.502 INFO:teuthology.orchestra.run.vm01.stdout: Found existing installation: xmltodict 0.12.0 2026-03-20T18:03:34.502 INFO:teuthology.orchestra.run.vm01.stdout: Not uninstalling xmltodict at /usr/lib/python3.9/site-packages, outside environment /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv 2026-03-20T18:03:34.502 INFO:teuthology.orchestra.run.vm01.stdout: Can't uninstall 'xmltodict'. No files were found to uninstall. 2026-03-20T18:03:34.543 INFO:teuthology.orchestra.run.vm01.stdout: Attempting uninstall: pika 2026-03-20T18:03:34.543 INFO:teuthology.orchestra.run.vm01.stdout: Found existing installation: pika 1.2.1 2026-03-20T18:03:34.543 INFO:teuthology.orchestra.run.vm01.stdout: Not uninstalling pika at /usr/lib/python3.9/site-packages, outside environment /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv 2026-03-20T18:03:34.543 INFO:teuthology.orchestra.run.vm01.stdout: Can't uninstall 'pika'. No files were found to uninstall. 2026-03-20T18:03:34.982 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:34.983 INFO:teuthology.orchestra.run.vm01.stdout:Successfully installed boto-2.49.0 boto3-1.42.72 botocore-1.42.72 cloudevents-1.12.1 configparser-7.2.0 deprecation-2.1.0 kafka-python-2.3.0 pika-1.3.2 pynose-1.5.5 s3transfer-0.16.0 xmltodict-1.0.4 2026-03-20T18:03:35.175 INFO:teuthology.orchestra.run.vm01.stdout:running develop 2026-03-20T18:03:35.207 INFO:teuthology.orchestra.run.vm01.stdout:running egg_info 2026-03-20T18:03:35.207 INFO:teuthology.orchestra.run.vm01.stdout:creating bn_tests.egg-info 2026-03-20T18:03:35.207 INFO:teuthology.orchestra.run.vm01.stdout:writing bn_tests.egg-info/PKG-INFO 2026-03-20T18:03:35.207 INFO:teuthology.orchestra.run.vm01.stdout:writing dependency_links to bn_tests.egg-info/dependency_links.txt 2026-03-20T18:03:35.207 INFO:teuthology.orchestra.run.vm01.stdout:writing requirements to bn_tests.egg-info/requires.txt 2026-03-20T18:03:35.207 INFO:teuthology.orchestra.run.vm01.stdout:writing top-level names to bn_tests.egg-info/top_level.txt 2026-03-20T18:03:35.207 INFO:teuthology.orchestra.run.vm01.stdout:writing manifest file 'bn_tests.egg-info/SOURCES.txt' 2026-03-20T18:03:35.208 INFO:teuthology.orchestra.run.vm01.stdout:reading manifest file 'bn_tests.egg-info/SOURCES.txt' 2026-03-20T18:03:35.208 INFO:teuthology.orchestra.run.vm01.stdout:writing manifest file 'bn_tests.egg-info/SOURCES.txt' 2026-03-20T18:03:35.209 INFO:teuthology.orchestra.run.vm01.stdout:running build_ext 2026-03-20T18:03:35.214 INFO:teuthology.orchestra.run.vm01.stdout:Creating /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.9/site-packages/bn-tests.egg-link (link to .) 2026-03-20T18:03:35.214 INFO:teuthology.orchestra.run.vm01.stdout:Adding bn-tests 0.0.1 to easy-install.pth file 2026-03-20T18:03:35.214 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:35.214 INFO:teuthology.orchestra.run.vm01.stdout:Installed /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification 2026-03-20T18:03:35.215 INFO:teuthology.orchestra.run.vm01.stdout:Processing dependencies for bn-tests==0.0.1 2026-03-20T18:03:35.222 INFO:teuthology.orchestra.run.vm01.stdout:Searching for boto3==1.42.72 2026-03-20T18:03:35.222 INFO:teuthology.orchestra.run.vm01.stdout:Best match: boto3 1.42.72 2026-03-20T18:03:35.222 INFO:teuthology.orchestra.run.vm01.stdout:Adding boto3 1.42.72 to easy-install.pth file 2026-03-20T18:03:35.222 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:35.222 INFO:teuthology.orchestra.run.vm01.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.9/site-packages 2026-03-20T18:03:35.223 INFO:teuthology.orchestra.run.vm01.stdout:Searching for boto==2.49.0 2026-03-20T18:03:35.223 INFO:teuthology.orchestra.run.vm01.stdout:Best match: boto 2.49.0 2026-03-20T18:03:35.223 INFO:teuthology.orchestra.run.vm01.stdout:Adding boto 2.49.0 to easy-install.pth file 2026-03-20T18:03:35.223 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:35.223 INFO:teuthology.orchestra.run.vm01.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.9/site-packages 2026-03-20T18:03:35.223 INFO:teuthology.orchestra.run.vm01.stdout:Searching for botocore==1.42.72 2026-03-20T18:03:35.223 INFO:teuthology.orchestra.run.vm01.stdout:Best match: botocore 1.42.72 2026-03-20T18:03:35.223 INFO:teuthology.orchestra.run.vm01.stdout:Adding botocore 1.42.72 to easy-install.pth file 2026-03-20T18:03:35.223 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:35.223 INFO:teuthology.orchestra.run.vm01.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.9/site-packages 2026-03-20T18:03:35.224 INFO:teuthology.orchestra.run.vm01.stdout:Searching for s3transfer==0.16.0 2026-03-20T18:03:35.224 INFO:teuthology.orchestra.run.vm01.stdout:Best match: s3transfer 0.16.0 2026-03-20T18:03:35.224 INFO:teuthology.orchestra.run.vm01.stdout:Adding s3transfer 0.16.0 to easy-install.pth file 2026-03-20T18:03:35.224 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:35.224 INFO:teuthology.orchestra.run.vm01.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.9/site-packages 2026-03-20T18:03:35.224 INFO:teuthology.orchestra.run.vm01.stdout:Searching for jmespath==1.0.1 2026-03-20T18:03:35.224 INFO:teuthology.orchestra.run.vm01.stdout:Best match: jmespath 1.0.1 2026-03-20T18:03:35.224 INFO:teuthology.orchestra.run.vm01.stdout:Adding jmespath 1.0.1 to easy-install.pth file 2026-03-20T18:03:35.224 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:35.224 INFO:teuthology.orchestra.run.vm01.stdout:Using /usr/lib/python3.9/site-packages 2026-03-20T18:03:35.225 INFO:teuthology.orchestra.run.vm01.stdout:Searching for urllib3==1.26.5 2026-03-20T18:03:35.225 INFO:teuthology.orchestra.run.vm01.stdout:Best match: urllib3 1.26.5 2026-03-20T18:03:35.225 INFO:teuthology.orchestra.run.vm01.stdout:Adding urllib3 1.26.5 to easy-install.pth file 2026-03-20T18:03:35.225 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:35.225 INFO:teuthology.orchestra.run.vm01.stdout:Using /usr/lib/python3.9/site-packages 2026-03-20T18:03:35.225 INFO:teuthology.orchestra.run.vm01.stdout:Searching for python-dateutil==2.9.0.post0 2026-03-20T18:03:35.225 INFO:teuthology.orchestra.run.vm01.stdout:Best match: python-dateutil 2.9.0.post0 2026-03-20T18:03:35.225 INFO:teuthology.orchestra.run.vm01.stdout:Adding python-dateutil 2.9.0.post0 to easy-install.pth file 2026-03-20T18:03:35.225 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:35.225 INFO:teuthology.orchestra.run.vm01.stdout:Using /usr/lib/python3.9/site-packages 2026-03-20T18:03:35.226 INFO:teuthology.orchestra.run.vm01.stdout:Searching for six==1.15.0 2026-03-20T18:03:35.226 INFO:teuthology.orchestra.run.vm01.stdout:Best match: six 1.15.0 2026-03-20T18:03:35.226 INFO:teuthology.orchestra.run.vm01.stdout:Adding six 1.15.0 to easy-install.pth file 2026-03-20T18:03:35.226 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:03:35.226 INFO:teuthology.orchestra.run.vm01.stdout:Using /usr/lib/python3.9/site-packages 2026-03-20T18:03:35.226 INFO:teuthology.orchestra.run.vm01.stdout:Finished processing dependencies for bn-tests==0.0.1 2026-03-20T18:03:35.240 INFO:tasks.notification_tests:Running bucket-notifications-tests... 2026-03-20T18:03:35.240 DEBUG:teuthology.orchestra.run.vm01:bucket notification tests against different endpoints> BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a amqp_test 2026-03-20T18:03:35.653 INFO:teuthology.orchestra.run.vm01.stderr:test s3 notification filter on master ... /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.9/site-packages/boto3/compat.py:89: PythonDeprecationWarning: Boto3 will no longer support Python 3.9 starting April 29, 2026. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.10 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2026-03-20T18:03:35.653 INFO:teuthology.orchestra.run.vm01.stderr: warnings.warn(warning, PythonDeprecationWarning) 2026-03-20T18:03:45.398 INFO:teuthology.orchestra.run.vm01.stderr:ok 2026-03-20T18:03:45.398 INFO:teuthology.orchestra.run.vm01.stderr:test pushing amqp notification ... ok 2026-03-20T18:03:53.550 INFO:teuthology.orchestra.run.vm01.stderr:bucket_notification.test_bn.test_ps_s3_creation_triggers_on_master ... ok 2026-03-20T18:03:53.921 INFO:teuthology.orchestra.run.vm01.stderr:test that uploads object using HTTP POST ... ok 2026-03-20T18:03:53.949 INFO:teuthology.orchestra.run.vm01.stderr:test ampq multipart object upload ... ok 2026-03-20T18:03:55.429 INFO:teuthology.orchestra.run.vm01.stderr:test notification of filtering metadata, ampq endpoint ... ignoring --setuser ceph since I am not root 2026-03-20T18:03:55.429 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:03:55.433 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.433+0000 7fa44ac10900 -1 asok(0x55a3041855c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.66841.asok': (13) Permission denied 2026-03-20T18:03:55.447 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.447+0000 7fa44ac10900 20 rados->read ofs=0 len=0 2026-03-20T18:03:55.447 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.448+0000 7fa44ac10900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:03:55.448 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.448+0000 7fa44ac10900 20 rados->read ofs=0 len=0 2026-03-20T18:03:55.448 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.449+0000 7fa44ac10900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:03:55.448 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.449+0000 7fa44ac10900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:03:55.448 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.449+0000 7fa44ac10900 20 rados->read ofs=0 len=0 2026-03-20T18:03:55.448 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.449+0000 7fa44ac10900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:03:55.448 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.449+0000 7fa44ac10900 20 rados->read ofs=0 len=0 2026-03-20T18:03:55.449 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.450+0000 7fa44ac10900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:03:55.449 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.450+0000 7fa44ac10900 20 rados->read ofs=0 len=0 2026-03-20T18:03:55.450 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.450+0000 7fa44ac10900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:03:55.450 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.451+0000 7fa44ac10900 20 rados->read ofs=0 len=0 2026-03-20T18:03:55.450 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.451+0000 7fa44ac10900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:03:55.450 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.451+0000 7fa44ac10900 20 rados->read ofs=0 len=0 2026-03-20T18:03:55.450 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.451+0000 7fa44ac10900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:03:55.450 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.451+0000 7fa44ac10900 20 rados->read ofs=0 len=0 2026-03-20T18:03:55.451 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.452+0000 7fa44ac10900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:03:55.451 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.452+0000 7fa44ac10900 20 rados->read ofs=0 len=0 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.452+0000 7fa44ac10900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.452+0000 7fa44ac10900 20 zone MyZone found 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.452+0000 7fa44ac10900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.452+0000 7fa44ac10900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.452+0000 7fa44ac10900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.452+0000 7fa44ac10900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.452+0000 7fa44ac10900 20 period zonegroup name MyZoneGroup 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.452+0000 7fa44ac10900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.452+0000 7fa44ac10900 20 rados->read ofs=0 len=0 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.453+0000 7fa44ac10900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.453+0000 7fa44ac10900 20 rados->read ofs=0 len=0 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.453+0000 7fa44ac10900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.453+0000 7fa44ac10900 20 started sync module instance, tier type = 2026-03-20T18:03:55.452 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.453+0000 7fa44ac10900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:03:55.456 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.457+0000 7fa44ac10900 20 add_watcher() i=3 2026-03-20T18:03:55.457 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.458+0000 7fa44ac10900 20 add_watcher() i=5 2026-03-20T18:03:55.457 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.458+0000 7fa44ac10900 20 add_watcher() i=6 2026-03-20T18:03:55.457 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.458+0000 7fa44ac10900 20 add_watcher() i=1 2026-03-20T18:03:55.457 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.458+0000 7fa44ac10900 20 add_watcher() i=2 2026-03-20T18:03:55.457 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.458+0000 7fa44ac10900 20 add_watcher() i=7 2026-03-20T18:03:55.457 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.458+0000 7fa44ac10900 20 add_watcher() i=0 2026-03-20T18:03:55.457 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.458+0000 7fa44ac10900 20 add_watcher() i=4 2026-03-20T18:03:55.457 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.458+0000 7fa44ac10900 2 all 8 watchers are set, enabling cache 2026-03-20T18:03:55.459 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.460+0000 7fa44ac10900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:03:55.459 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.460+0000 7fa44ac10900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:03:55.459 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.460+0000 7fa44ac10900 5 note: GC not initialized 2026-03-20T18:03:55.460 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.460+0000 7fa3f8fe1640 20 reqs_thread_entry: start 2026-03-20T18:03:55.496 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.496+0000 7fa44ac10900 20 init_complete bucket index max shards: 11 2026-03-20T18:03:55.496 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.496+0000 7fa44ac10900 20 Filter name: none 2026-03-20T18:03:55.496 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.497+0000 7fa3aa7fc640 20 reqs_thread_entry: start 2026-03-20T18:03:55.496 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.497+0000 7fa44ac10900 20 rados->read ofs=0 len=0 2026-03-20T18:03:55.496 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.497+0000 7fa44ac10900 20 rados_obj.operate() r=0 bl.length=260 2026-03-20T18:03:55.500 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.501+0000 7fa44ac10900 20 remove_watcher() i=3 2026-03-20T18:03:55.500 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.501+0000 7fa44ac10900 2 removed watcher, disabling cache 2026-03-20T18:03:55.500 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.501+0000 7fa44ac10900 20 remove_watcher() i=2 2026-03-20T18:03:55.501 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.501+0000 7fa44ac10900 20 remove_watcher() i=0 2026-03-20T18:03:55.501 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.502+0000 7fa44ac10900 20 remove_watcher() i=7 2026-03-20T18:03:55.501 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.502+0000 7fa44ac10900 20 remove_watcher() i=4 2026-03-20T18:03:55.501 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.502+0000 7fa44ac10900 20 remove_watcher() i=6 2026-03-20T18:03:55.501 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.502+0000 7fa44ac10900 20 remove_watcher() i=5 2026-03-20T18:03:55.501 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:03:55.502+0000 7fa44ac10900 20 remove_watcher() i=1 2026-03-20T18:04:00.541 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:04:00.541 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:04:00.545 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.546+0000 7f1957e22900 -1 asok(0x55d19fa9b5c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.66939.asok': (13) Permission denied 2026-03-20T18:04:00.564 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.565+0000 7f1957e22900 20 rados->read ofs=0 len=0 2026-03-20T18:04:00.565 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.566+0000 7f1957e22900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:00.565 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.566+0000 7f1957e22900 20 rados->read ofs=0 len=0 2026-03-20T18:04:00.565 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.566+0000 7f1957e22900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:00.566 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.566+0000 7f1957e22900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:04:00.566 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.566+0000 7f1957e22900 20 rados->read ofs=0 len=0 2026-03-20T18:04:00.566 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.566+0000 7f1957e22900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:00.566 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.566+0000 7f1957e22900 20 rados->read ofs=0 len=0 2026-03-20T18:04:00.566 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.567+0000 7f1957e22900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:04:00.566 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.567+0000 7f1957e22900 20 rados->read ofs=0 len=0 2026-03-20T18:04:00.567 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.568+0000 7f1957e22900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:04:00.567 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.568+0000 7f1957e22900 20 rados->read ofs=0 len=0 2026-03-20T18:04:00.567 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.568+0000 7f1957e22900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:00.567 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.568+0000 7f1957e22900 20 rados->read ofs=0 len=0 2026-03-20T18:04:00.567 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.568+0000 7f1957e22900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:00.567 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.568+0000 7f1957e22900 20 rados->read ofs=0 len=0 2026-03-20T18:04:00.568 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.569+0000 7f1957e22900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:00.568 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.569+0000 7f1957e22900 20 rados->read ofs=0 len=0 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.569+0000 7f1957e22900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.569+0000 7f1957e22900 20 zone MyZone found 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.569+0000 7f1957e22900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.569+0000 7f1957e22900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.569+0000 7f1957e22900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.569+0000 7f1957e22900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.569+0000 7f1957e22900 20 period zonegroup name MyZoneGroup 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.569+0000 7f1957e22900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.569+0000 7f1957e22900 20 rados->read ofs=0 len=0 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.570+0000 7f1957e22900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.570+0000 7f1957e22900 20 rados->read ofs=0 len=0 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.570+0000 7f1957e22900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.570+0000 7f1957e22900 20 started sync module instance, tier type = 2026-03-20T18:04:00.569 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.570+0000 7f1957e22900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:04:00.573 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.574+0000 7f1957e22900 20 add_watcher() i=3 2026-03-20T18:04:00.573 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.574+0000 7f1957e22900 20 add_watcher() i=2 2026-03-20T18:04:00.573 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.574+0000 7f1957e22900 20 add_watcher() i=5 2026-03-20T18:04:00.574 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.574+0000 7f1957e22900 20 add_watcher() i=6 2026-03-20T18:04:00.574 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.575+0000 7f1957e22900 20 add_watcher() i=0 2026-03-20T18:04:00.574 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.575+0000 7f1957e22900 20 add_watcher() i=7 2026-03-20T18:04:00.574 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.575+0000 7f1957e22900 20 add_watcher() i=1 2026-03-20T18:04:00.575 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.575+0000 7f1957e22900 20 add_watcher() i=4 2026-03-20T18:04:00.575 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.575+0000 7f1957e22900 2 all 8 watchers are set, enabling cache 2026-03-20T18:04:00.576 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.577+0000 7f1957e22900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:04:00.576 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.577+0000 7f1957e22900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:04:00.576 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.577+0000 7f1957e22900 5 note: GC not initialized 2026-03-20T18:04:00.577 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.577+0000 7f19057e2640 20 reqs_thread_entry: start 2026-03-20T18:04:00.621 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.621+0000 7f1957e22900 20 init_complete bucket index max shards: 11 2026-03-20T18:04:00.621 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.621+0000 7f1957e22900 20 Filter name: none 2026-03-20T18:04:00.621 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.621+0000 7f18b6ffd640 20 reqs_thread_entry: start 2026-03-20T18:04:00.621 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.622+0000 7f1957e22900 20 rados->read ofs=0 len=0 2026-03-20T18:04:00.622 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.622+0000 7f1957e22900 20 rados_obj.operate() r=0 bl.length=260 2026-03-20T18:04:00.627 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.628+0000 7f1957e22900 20 remove_watcher() i=0 2026-03-20T18:04:00.627 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.628+0000 7f1957e22900 2 removed watcher, disabling cache 2026-03-20T18:04:00.627 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.628+0000 7f1957e22900 20 remove_watcher() i=4 2026-03-20T18:04:00.627 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.628+0000 7f1957e22900 20 remove_watcher() i=3 2026-03-20T18:04:00.628 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.629+0000 7f1957e22900 20 remove_watcher() i=7 2026-03-20T18:04:00.628 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.629+0000 7f1957e22900 20 remove_watcher() i=5 2026-03-20T18:04:00.628 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.629+0000 7f1957e22900 20 remove_watcher() i=6 2026-03-20T18:04:00.628 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.629+0000 7f1957e22900 20 remove_watcher() i=2 2026-03-20T18:04:00.628 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:00.629+0000 7f1957e22900 20 remove_watcher() i=1 2026-03-20T18:04:05.669 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:04:05.669 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:04:05.673 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.674+0000 7f94bd12d900 -1 asok(0x556e697575c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.67037.asok': (13) Permission denied 2026-03-20T18:04:05.690 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.691+0000 7f94bd12d900 20 rados->read ofs=0 len=0 2026-03-20T18:04:05.691 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.692+0000 7f94bd12d900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:05.691 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.692+0000 7f94bd12d900 20 rados->read ofs=0 len=0 2026-03-20T18:04:05.692 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.693+0000 7f94bd12d900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:05.692 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.693+0000 7f94bd12d900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:04:05.692 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.693+0000 7f94bd12d900 20 rados->read ofs=0 len=0 2026-03-20T18:04:05.692 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.693+0000 7f94bd12d900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:05.692 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.693+0000 7f94bd12d900 20 rados->read ofs=0 len=0 2026-03-20T18:04:05.693 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.694+0000 7f94bd12d900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:04:05.693 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.694+0000 7f94bd12d900 20 rados->read ofs=0 len=0 2026-03-20T18:04:05.694 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.694+0000 7f94bd12d900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:04:05.694 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.694+0000 7f94bd12d900 20 rados->read ofs=0 len=0 2026-03-20T18:04:05.694 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.695+0000 7f94bd12d900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:05.694 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.695+0000 7f94bd12d900 20 rados->read ofs=0 len=0 2026-03-20T18:04:05.694 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.695+0000 7f94bd12d900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:05.694 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.695+0000 7f94bd12d900 20 rados->read ofs=0 len=0 2026-03-20T18:04:05.695 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.695+0000 7f94bd12d900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:05.695 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.695+0000 7f94bd12d900 20 rados->read ofs=0 len=0 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.696+0000 7f94bd12d900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.696+0000 7f94bd12d900 20 zone MyZone found 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.696+0000 7f94bd12d900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.696+0000 7f94bd12d900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.696+0000 7f94bd12d900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.696+0000 7f94bd12d900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.696+0000 7f94bd12d900 20 period zonegroup name MyZoneGroup 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.696+0000 7f94bd12d900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.696+0000 7f94bd12d900 20 rados->read ofs=0 len=0 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.697+0000 7f94bd12d900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.697+0000 7f94bd12d900 20 rados->read ofs=0 len=0 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.697+0000 7f94bd12d900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.697+0000 7f94bd12d900 20 started sync module instance, tier type = 2026-03-20T18:04:05.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.697+0000 7f94bd12d900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:04:05.700 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.701+0000 7f94bd12d900 20 add_watcher() i=3 2026-03-20T18:04:05.701 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.701+0000 7f94bd12d900 20 add_watcher() i=6 2026-03-20T18:04:05.701 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.701+0000 7f94bd12d900 20 add_watcher() i=5 2026-03-20T18:04:05.701 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.701+0000 7f94bd12d900 20 add_watcher() i=1 2026-03-20T18:04:05.701 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.701+0000 7f94bd12d900 20 add_watcher() i=7 2026-03-20T18:04:05.701 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.702+0000 7f94bd12d900 20 add_watcher() i=2 2026-03-20T18:04:05.701 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.702+0000 7f94bd12d900 20 add_watcher() i=0 2026-03-20T18:04:05.702 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.702+0000 7f94bd12d900 20 add_watcher() i=4 2026-03-20T18:04:05.702 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.702+0000 7f94bd12d900 2 all 8 watchers are set, enabling cache 2026-03-20T18:04:05.704 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.705+0000 7f94bd12d900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:04:05.704 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.705+0000 7f94bd12d900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:04:05.704 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.705+0000 7f94bd12d900 5 note: GC not initialized 2026-03-20T18:04:05.704 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.705+0000 7f946afe5640 20 reqs_thread_entry: start 2026-03-20T18:04:05.745 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.745+0000 7f94bd12d900 20 init_complete bucket index max shards: 11 2026-03-20T18:04:05.745 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.745+0000 7f94bd12d900 20 Filter name: none 2026-03-20T18:04:05.745 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.745+0000 7f9468fe1640 20 reqs_thread_entry: start 2026-03-20T18:04:05.745 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.746+0000 7f94bd12d900 20 rados->read ofs=0 len=0 2026-03-20T18:04:05.745 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.746+0000 7f94bd12d900 20 rados_obj.operate() r=0 bl.length=260 2026-03-20T18:04:05.750 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.750+0000 7f94bd12d900 20 remove_watcher() i=0 2026-03-20T18:04:05.750 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.750+0000 7f94bd12d900 2 removed watcher, disabling cache 2026-03-20T18:04:05.750 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.750+0000 7f94bd12d900 20 remove_watcher() i=3 2026-03-20T18:04:05.750 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.750+0000 7f94bd12d900 20 remove_watcher() i=2 2026-03-20T18:04:05.750 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.751+0000 7f94bd12d900 20 remove_watcher() i=7 2026-03-20T18:04:05.750 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.751+0000 7f94bd12d900 20 remove_watcher() i=5 2026-03-20T18:04:05.750 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.751+0000 7f94bd12d900 20 remove_watcher() i=6 2026-03-20T18:04:05.750 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.751+0000 7f94bd12d900 20 remove_watcher() i=4 2026-03-20T18:04:05.750 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:05.751+0000 7f94bd12d900 20 remove_watcher() i=1 2026-03-20T18:04:10.787 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:04:10.787 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:04:10.791 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.791+0000 7fe6cb324900 -1 asok(0x5584df6925c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.67133.asok': (13) Permission denied 2026-03-20T18:04:10.808 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.808+0000 7fe6cb324900 20 rados->read ofs=0 len=0 2026-03-20T18:04:10.808 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.809+0000 7fe6cb324900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:10.808 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.809+0000 7fe6cb324900 20 rados->read ofs=0 len=0 2026-03-20T18:04:10.809 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.810+0000 7fe6cb324900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:10.809 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.810+0000 7fe6cb324900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:04:10.809 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.810+0000 7fe6cb324900 20 rados->read ofs=0 len=0 2026-03-20T18:04:10.809 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.810+0000 7fe6cb324900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:10.809 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.810+0000 7fe6cb324900 20 rados->read ofs=0 len=0 2026-03-20T18:04:10.810 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.811+0000 7fe6cb324900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:04:10.810 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.811+0000 7fe6cb324900 20 rados->read ofs=0 len=0 2026-03-20T18:04:10.811 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.812+0000 7fe6cb324900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:04:10.811 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.812+0000 7fe6cb324900 20 rados->read ofs=0 len=0 2026-03-20T18:04:10.811 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.812+0000 7fe6cb324900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:10.811 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.812+0000 7fe6cb324900 20 rados->read ofs=0 len=0 2026-03-20T18:04:10.811 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.812+0000 7fe6cb324900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:10.811 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.812+0000 7fe6cb324900 20 rados->read ofs=0 len=0 2026-03-20T18:04:10.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.813+0000 7fe6cb324900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:10.812 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.813+0000 7fe6cb324900 20 rados->read ofs=0 len=0 2026-03-20T18:04:10.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:04:10.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 20 zone MyZone found 2026-03-20T18:04:10.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:04:10.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:04:10.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:04:10.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:04:10.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 20 period zonegroup name MyZoneGroup 2026-03-20T18:04:10.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:04:10.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 20 rados->read ofs=0 len=0 2026-03-20T18:04:10.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:04:10.813 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 20 rados->read ofs=0 len=0 2026-03-20T18:04:10.814 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:04:10.814 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 20 started sync module instance, tier type = 2026-03-20T18:04:10.814 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.814+0000 7fe6cb324900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:04:10.817 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.818+0000 7fe6cb324900 20 add_watcher() i=3 2026-03-20T18:04:10.817 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.818+0000 7fe6cb324900 20 add_watcher() i=6 2026-03-20T18:04:10.818 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.818+0000 7fe6cb324900 20 add_watcher() i=2 2026-03-20T18:04:10.818 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.818+0000 7fe6cb324900 20 add_watcher() i=5 2026-03-20T18:04:10.818 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.819+0000 7fe6cb324900 20 add_watcher() i=1 2026-03-20T18:04:10.818 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.819+0000 7fe6cb324900 20 add_watcher() i=0 2026-03-20T18:04:10.818 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.819+0000 7fe6cb324900 20 add_watcher() i=4 2026-03-20T18:04:10.818 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.819+0000 7fe6cb324900 20 add_watcher() i=7 2026-03-20T18:04:10.818 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.819+0000 7fe6cb324900 2 all 8 watchers are set, enabling cache 2026-03-20T18:04:10.820 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.820+0000 7fe6cb324900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:04:10.820 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.820+0000 7fe6cb324900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:04:10.820 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.820+0000 7fe6cb324900 5 note: GC not initialized 2026-03-20T18:04:10.820 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.821+0000 7fe67a7dc640 20 reqs_thread_entry: start 2026-03-20T18:04:10.851 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.852+0000 7fe6cb324900 20 init_complete bucket index max shards: 11 2026-03-20T18:04:10.851 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.852+0000 7fe6cb324900 20 Filter name: none 2026-03-20T18:04:10.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.852+0000 7fe62bfff640 20 reqs_thread_entry: start 2026-03-20T18:04:10.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.853+0000 7fe6cb324900 20 rados->read ofs=0 len=0 2026-03-20T18:04:10.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.853+0000 7fe6cb324900 20 rados_obj.operate() r=0 bl.length=260 2026-03-20T18:04:10.856 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.856+0000 7fe6cb324900 20 remove_watcher() i=0 2026-03-20T18:04:10.856 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.856+0000 7fe6cb324900 2 removed watcher, disabling cache 2026-03-20T18:04:10.856 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.857+0000 7fe6cb324900 20 remove_watcher() i=4 2026-03-20T18:04:10.856 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.857+0000 7fe6cb324900 20 remove_watcher() i=3 2026-03-20T18:04:10.856 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.857+0000 7fe6cb324900 20 remove_watcher() i=6 2026-03-20T18:04:10.856 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.857+0000 7fe6cb324900 20 remove_watcher() i=5 2026-03-20T18:04:10.856 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.857+0000 7fe6cb324900 20 remove_watcher() i=1 2026-03-20T18:04:10.856 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.857+0000 7fe6cb324900 20 remove_watcher() i=7 2026-03-20T18:04:10.856 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:10.857+0000 7fe6cb324900 20 remove_watcher() i=2 2026-03-20T18:04:15.895 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:04:15.895 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:04:15.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.900+0000 7f87fd31f900 -1 asok(0x55f7f4c245c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.67232.asok': (13) Permission denied 2026-03-20T18:04:15.916 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.916+0000 7f87fd31f900 20 rados->read ofs=0 len=0 2026-03-20T18:04:15.917 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.918+0000 7f87fd31f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:15.917 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.918+0000 7f87fd31f900 20 rados->read ofs=0 len=0 2026-03-20T18:04:15.918 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.918+0000 7f87fd31f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:15.918 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.918+0000 7f87fd31f900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:04:15.918 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.918+0000 7f87fd31f900 20 rados->read ofs=0 len=0 2026-03-20T18:04:15.918 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.919+0000 7f87fd31f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:15.918 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.919+0000 7f87fd31f900 20 rados->read ofs=0 len=0 2026-03-20T18:04:15.919 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.919+0000 7f87fd31f900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:04:15.919 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.919+0000 7f87fd31f900 20 rados->read ofs=0 len=0 2026-03-20T18:04:15.919 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.920+0000 7f87fd31f900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:04:15.920 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.920+0000 7f87fd31f900 20 rados->read ofs=0 len=0 2026-03-20T18:04:15.920 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.920+0000 7f87fd31f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:15.920 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.920+0000 7f87fd31f900 20 rados->read ofs=0 len=0 2026-03-20T18:04:15.920 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.921+0000 7f87fd31f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:15.920 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.921+0000 7f87fd31f900 20 rados->read ofs=0 len=0 2026-03-20T18:04:15.921 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.921+0000 7f87fd31f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:15.921 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.921+0000 7f87fd31f900 20 rados->read ofs=0 len=0 2026-03-20T18:04:15.921 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.922+0000 7f87fd31f900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:04:15.921 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.922+0000 7f87fd31f900 20 zone MyZone found 2026-03-20T18:04:15.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.922+0000 7f87fd31f900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:04:15.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.922+0000 7f87fd31f900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:04:15.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.922+0000 7f87fd31f900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:04:15.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.922+0000 7f87fd31f900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:04:15.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.922+0000 7f87fd31f900 20 period zonegroup name MyZoneGroup 2026-03-20T18:04:15.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.922+0000 7f87fd31f900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:04:15.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.922+0000 7f87fd31f900 20 rados->read ofs=0 len=0 2026-03-20T18:04:15.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.922+0000 7f87fd31f900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:04:15.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.922+0000 7f87fd31f900 20 rados->read ofs=0 len=0 2026-03-20T18:04:15.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.923+0000 7f87fd31f900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:04:15.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.923+0000 7f87fd31f900 20 started sync module instance, tier type = 2026-03-20T18:04:15.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.923+0000 7f87fd31f900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:04:15.926 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.927+0000 7f87fd31f900 20 add_watcher() i=2 2026-03-20T18:04:15.926 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.927+0000 7f87fd31f900 20 add_watcher() i=3 2026-03-20T18:04:15.927 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.927+0000 7f87fd31f900 20 add_watcher() i=0 2026-03-20T18:04:15.927 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.927+0000 7f87fd31f900 20 add_watcher() i=1 2026-03-20T18:04:15.927 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.927+0000 7f87fd31f900 20 add_watcher() i=5 2026-03-20T18:04:15.927 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.927+0000 7f87fd31f900 20 add_watcher() i=7 2026-03-20T18:04:15.927 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.928+0000 7f87fd31f900 20 add_watcher() i=4 2026-03-20T18:04:15.927 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.928+0000 7f87fd31f900 20 add_watcher() i=6 2026-03-20T18:04:15.927 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.928+0000 7f87fd31f900 2 all 8 watchers are set, enabling cache 2026-03-20T18:04:15.929 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.930+0000 7f87fd31f900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:04:15.929 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.930+0000 7f87fd31f900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:04:15.929 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.930+0000 7f87fd31f900 5 note: GC not initialized 2026-03-20T18:04:15.929 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.930+0000 7f87a77fe640 20 reqs_thread_entry: start 2026-03-20T18:04:15.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.972+0000 7f87fd31f900 20 init_complete bucket index max shards: 11 2026-03-20T18:04:15.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.972+0000 7f87fd31f900 20 Filter name: none 2026-03-20T18:04:15.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.972+0000 7f87a57fa640 20 reqs_thread_entry: start 2026-03-20T18:04:15.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.973+0000 7f87fd31f900 20 rados->read ofs=0 len=0 2026-03-20T18:04:15.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.973+0000 7f87fd31f900 20 rados_obj.operate() r=0 bl.length=260 2026-03-20T18:04:15.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.977+0000 7f87fd31f900 20 remove_watcher() i=7 2026-03-20T18:04:15.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.977+0000 7f87fd31f900 2 removed watcher, disabling cache 2026-03-20T18:04:15.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.977+0000 7f87fd31f900 20 remove_watcher() i=3 2026-03-20T18:04:15.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.977+0000 7f87fd31f900 20 remove_watcher() i=0 2026-03-20T18:04:15.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.977+0000 7f87fd31f900 20 remove_watcher() i=4 2026-03-20T18:04:15.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.978+0000 7f87fd31f900 20 remove_watcher() i=1 2026-03-20T18:04:15.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.978+0000 7f87fd31f900 20 remove_watcher() i=5 2026-03-20T18:04:15.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.978+0000 7f87fd31f900 20 remove_watcher() i=2 2026-03-20T18:04:15.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:15.978+0000 7f87fd31f900 20 remove_watcher() i=6 2026-03-20T18:04:21.102 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:04:21.103 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:04:21.106 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.107+0000 7f5d6b4f8900 -1 asok(0x55d898c0a5c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.67330.asok': (13) Permission denied 2026-03-20T18:04:21.121 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.122+0000 7f5d6b4f8900 20 rados->read ofs=0 len=0 2026-03-20T18:04:21.122 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.123+0000 7f5d6b4f8900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:21.122 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.123+0000 7f5d6b4f8900 20 rados->read ofs=0 len=0 2026-03-20T18:04:21.123 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.123+0000 7f5d6b4f8900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:21.123 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.123+0000 7f5d6b4f8900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:04:21.123 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.123+0000 7f5d6b4f8900 20 rados->read ofs=0 len=0 2026-03-20T18:04:21.123 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.124+0000 7f5d6b4f8900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:21.123 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.124+0000 7f5d6b4f8900 20 rados->read ofs=0 len=0 2026-03-20T18:04:21.124 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.124+0000 7f5d6b4f8900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:04:21.124 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.124+0000 7f5d6b4f8900 20 rados->read ofs=0 len=0 2026-03-20T18:04:21.124 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.125+0000 7f5d6b4f8900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:04:21.124 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.125+0000 7f5d6b4f8900 20 rados->read ofs=0 len=0 2026-03-20T18:04:21.124 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.125+0000 7f5d6b4f8900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:21.124 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.125+0000 7f5d6b4f8900 20 rados->read ofs=0 len=0 2026-03-20T18:04:21.125 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.125+0000 7f5d6b4f8900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:21.125 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.125+0000 7f5d6b4f8900 20 rados->read ofs=0 len=0 2026-03-20T18:04:21.125 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.126+0000 7f5d6b4f8900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:21.125 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.126+0000 7f5d6b4f8900 20 rados->read ofs=0 len=0 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 20 zone MyZone found 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 20 period zonegroup name MyZoneGroup 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 20 rados->read ofs=0 len=0 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 20 rados->read ofs=0 len=0 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:04:21.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 20 started sync module instance, tier type = 2026-03-20T18:04:21.127 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.127+0000 7f5d6b4f8900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:04:21.131 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.132+0000 7f5d6b4f8900 20 add_watcher() i=5 2026-03-20T18:04:21.131 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.132+0000 7f5d6b4f8900 20 add_watcher() i=0 2026-03-20T18:04:21.131 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.132+0000 7f5d6b4f8900 20 add_watcher() i=3 2026-03-20T18:04:21.131 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.132+0000 7f5d6b4f8900 20 add_watcher() i=6 2026-03-20T18:04:21.131 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.132+0000 7f5d6b4f8900 20 add_watcher() i=2 2026-03-20T18:04:21.131 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.132+0000 7f5d6b4f8900 20 add_watcher() i=7 2026-03-20T18:04:21.132 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.133+0000 7f5d6b4f8900 20 add_watcher() i=1 2026-03-20T18:04:21.132 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.133+0000 7f5d6b4f8900 20 add_watcher() i=4 2026-03-20T18:04:21.132 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.133+0000 7f5d6b4f8900 2 all 8 watchers are set, enabling cache 2026-03-20T18:04:21.134 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.135+0000 7f5d6b4f8900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:04:21.134 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.135+0000 7f5d6b4f8900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:04:21.134 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.135+0000 7f5d6b4f8900 5 note: GC not initialized 2026-03-20T18:04:21.135 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.135+0000 7f5d16ffd640 20 reqs_thread_entry: start 2026-03-20T18:04:21.171 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.172+0000 7f5d6b4f8900 20 init_complete bucket index max shards: 11 2026-03-20T18:04:21.171 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.172+0000 7f5d6b4f8900 20 Filter name: none 2026-03-20T18:04:21.171 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.172+0000 7f5d14ff9640 20 reqs_thread_entry: start 2026-03-20T18:04:21.171 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.172+0000 7f5d6b4f8900 20 rados->read ofs=0 len=0 2026-03-20T18:04:21.172 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.173+0000 7f5d6b4f8900 20 rados_obj.operate() r=0 bl.length=260 2026-03-20T18:04:21.176 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.176+0000 7f5d6b4f8900 20 remove_watcher() i=0 2026-03-20T18:04:21.176 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.176+0000 7f5d6b4f8900 2 removed watcher, disabling cache 2026-03-20T18:04:21.176 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.176+0000 7f5d6b4f8900 20 remove_watcher() i=2 2026-03-20T18:04:21.176 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.177+0000 7f5d6b4f8900 20 remove_watcher() i=7 2026-03-20T18:04:21.176 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.177+0000 7f5d6b4f8900 20 remove_watcher() i=5 2026-03-20T18:04:21.176 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.177+0000 7f5d6b4f8900 20 remove_watcher() i=6 2026-03-20T18:04:21.176 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.177+0000 7f5d6b4f8900 20 remove_watcher() i=4 2026-03-20T18:04:21.176 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.177+0000 7f5d6b4f8900 20 remove_watcher() i=1 2026-03-20T18:04:21.176 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:21.177+0000 7f5d6b4f8900 20 remove_watcher() i=3 2026-03-20T18:04:26.213 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:04:26.213 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:04:26.217 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.217+0000 7fe7cab36900 -1 asok(0x5570326515c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.67427.asok': (13) Permission denied 2026-03-20T18:04:26.232 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.233+0000 7fe7cab36900 20 rados->read ofs=0 len=0 2026-03-20T18:04:26.233 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.234+0000 7fe7cab36900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:26.233 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.234+0000 7fe7cab36900 20 rados->read ofs=0 len=0 2026-03-20T18:04:26.233 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.234+0000 7fe7cab36900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:26.234 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.234+0000 7fe7cab36900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:04:26.234 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.234+0000 7fe7cab36900 20 rados->read ofs=0 len=0 2026-03-20T18:04:26.234 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.234+0000 7fe7cab36900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:26.234 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.235+0000 7fe7cab36900 20 rados->read ofs=0 len=0 2026-03-20T18:04:26.234 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.235+0000 7fe7cab36900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:04:26.235 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.235+0000 7fe7cab36900 20 rados->read ofs=0 len=0 2026-03-20T18:04:26.235 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.236+0000 7fe7cab36900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:04:26.235 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.236+0000 7fe7cab36900 20 rados->read ofs=0 len=0 2026-03-20T18:04:26.235 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.236+0000 7fe7cab36900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:26.235 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.236+0000 7fe7cab36900 20 rados->read ofs=0 len=0 2026-03-20T18:04:26.236 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.236+0000 7fe7cab36900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:04:26.236 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.236+0000 7fe7cab36900 20 rados->read ofs=0 len=0 2026-03-20T18:04:26.236 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.237+0000 7fe7cab36900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:04:26.236 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.237+0000 7fe7cab36900 20 rados->read ofs=0 len=0 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 20 zone MyZone found 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 20 period zonegroup name MyZoneGroup 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 20 rados->read ofs=0 len=0 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 20 rados->read ofs=0 len=0 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 20 started sync module instance, tier type = 2026-03-20T18:04:26.237 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.238+0000 7fe7cab36900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:04:26.241 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.241+0000 7fe7cab36900 20 add_watcher() i=3 2026-03-20T18:04:26.241 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.242+0000 7fe7cab36900 20 add_watcher() i=5 2026-03-20T18:04:26.241 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.242+0000 7fe7cab36900 20 add_watcher() i=6 2026-03-20T18:04:26.241 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.242+0000 7fe7cab36900 20 add_watcher() i=7 2026-03-20T18:04:26.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.243+0000 7fe7cab36900 20 add_watcher() i=2 2026-03-20T18:04:26.242 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.243+0000 7fe7cab36900 20 add_watcher() i=1 2026-03-20T18:04:26.243 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.244+0000 7fe7cab36900 20 add_watcher() i=4 2026-03-20T18:04:26.243 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.244+0000 7fe7cab36900 20 add_watcher() i=0 2026-03-20T18:04:26.243 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.244+0000 7fe7cab36900 2 all 8 watchers are set, enabling cache 2026-03-20T18:04:26.245 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.246+0000 7fe7cab36900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:04:26.245 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.246+0000 7fe7cab36900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:04:26.245 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.246+0000 7fe7cab36900 5 note: GC not initialized 2026-03-20T18:04:26.245 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.246+0000 7fe77a7dc640 20 reqs_thread_entry: start 2026-03-20T18:04:26.279 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.279+0000 7fe7cab36900 20 init_complete bucket index max shards: 11 2026-03-20T18:04:26.279 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.280+0000 7fe7cab36900 20 Filter name: none 2026-03-20T18:04:26.279 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.280+0000 7fe72bfff640 20 reqs_thread_entry: start 2026-03-20T18:04:26.279 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.280+0000 7fe7cab36900 20 rados->read ofs=0 len=0 2026-03-20T18:04:26.280 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.280+0000 7fe7cab36900 20 rados_obj.operate() r=0 bl.length=260 2026-03-20T18:04:26.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.283+0000 7fe7cab36900 20 remove_watcher() i=2 2026-03-20T18:04:26.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.283+0000 7fe7cab36900 2 removed watcher, disabling cache 2026-03-20T18:04:26.284 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.284+0000 7fe7cab36900 20 remove_watcher() i=5 2026-03-20T18:04:26.284 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.284+0000 7fe7cab36900 20 remove_watcher() i=7 2026-03-20T18:04:26.284 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.284+0000 7fe7cab36900 20 remove_watcher() i=3 2026-03-20T18:04:26.284 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.284+0000 7fe7cab36900 20 remove_watcher() i=6 2026-03-20T18:04:26.284 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.284+0000 7fe7cab36900 20 remove_watcher() i=0 2026-03-20T18:04:26.284 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.284+0000 7fe7cab36900 20 remove_watcher() i=1 2026-03-20T18:04:26.284 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:04:26.284+0000 7fe7cab36900 20 remove_watcher() i=4 2026-03-20T18:04:31.435 INFO:teuthology.orchestra.run.vm01.stderr:ok 2026-03-20T18:04:51.580 INFO:teuthology.orchestra.run.vm01.stderr:test s3 notification of metadata on master ... ok 2026-03-20T18:05:02.924 INFO:teuthology.orchestra.run.vm01.stderr:test s3 notification of tags on master ... ok 2026-03-20T18:05:08.491 INFO:teuthology.orchestra.run.vm01.stderr:test s3 notification of object versions ... ok 2026-03-20T18:05:15.559 INFO:teuthology.orchestra.run.vm01.stderr:test s3 notification of deletion markers on master ... ok 2026-03-20T18:05:15.672 INFO:teuthology.orchestra.run.vm01.stderr:test persistent topic stats, amqp endpoint ... ignoring --setuser ceph since I am not root 2026-03-20T18:05:15.673 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:05:15.677 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.677+0000 7f4bb6da2900 -1 asok(0x55deebe755c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.67552.asok': (13) Permission denied 2026-03-20T18:05:15.692 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.693+0000 7f4bb6da2900 20 rados->read ofs=0 len=0 2026-03-20T18:05:15.694 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.695+0000 7f4bb6da2900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:15.694 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.695+0000 7f4bb6da2900 20 rados->read ofs=0 len=0 2026-03-20T18:05:15.694 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.695+0000 7f4bb6da2900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:15.694 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.695+0000 7f4bb6da2900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:05:15.694 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.695+0000 7f4bb6da2900 20 rados->read ofs=0 len=0 2026-03-20T18:05:15.695 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.695+0000 7f4bb6da2900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:15.695 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.695+0000 7f4bb6da2900 20 rados->read ofs=0 len=0 2026-03-20T18:05:15.695 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.696+0000 7f4bb6da2900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:05:15.695 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.696+0000 7f4bb6da2900 20 rados->read ofs=0 len=0 2026-03-20T18:05:15.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.697+0000 7f4bb6da2900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:05:15.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.697+0000 7f4bb6da2900 20 rados->read ofs=0 len=0 2026-03-20T18:05:15.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.697+0000 7f4bb6da2900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:15.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.697+0000 7f4bb6da2900 20 rados->read ofs=0 len=0 2026-03-20T18:05:15.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.697+0000 7f4bb6da2900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:15.696 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.697+0000 7f4bb6da2900 20 rados->read ofs=0 len=0 2026-03-20T18:05:15.697 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.698+0000 7f4bb6da2900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:15.697 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.698+0000 7f4bb6da2900 20 rados->read ofs=0 len=0 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.698+0000 7f4bb6da2900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.698+0000 7f4bb6da2900 20 zone MyZone found 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.698+0000 7f4bb6da2900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.698+0000 7f4bb6da2900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.698+0000 7f4bb6da2900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.698+0000 7f4bb6da2900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.698+0000 7f4bb6da2900 20 period zonegroup name MyZoneGroup 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.698+0000 7f4bb6da2900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.698+0000 7f4bb6da2900 20 rados->read ofs=0 len=0 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.699+0000 7f4bb6da2900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.699+0000 7f4bb6da2900 20 rados->read ofs=0 len=0 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.699+0000 7f4bb6da2900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.699+0000 7f4bb6da2900 20 started sync module instance, tier type = 2026-03-20T18:05:15.698 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.699+0000 7f4bb6da2900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:05:15.703 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.703+0000 7f4bb6da2900 20 add_watcher() i=3 2026-03-20T18:05:15.703 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.703+0000 7f4bb6da2900 20 add_watcher() i=2 2026-03-20T18:05:15.703 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.703+0000 7f4bb6da2900 20 add_watcher() i=7 2026-03-20T18:05:15.703 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.704+0000 7f4bb6da2900 20 add_watcher() i=6 2026-03-20T18:05:15.703 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.704+0000 7f4bb6da2900 20 add_watcher() i=5 2026-03-20T18:05:15.704 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.705+0000 7f4bb6da2900 20 add_watcher() i=1 2026-03-20T18:05:15.705 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.705+0000 7f4bb6da2900 20 add_watcher() i=0 2026-03-20T18:05:15.705 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.705+0000 7f4bb6da2900 20 add_watcher() i=4 2026-03-20T18:05:15.705 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.705+0000 7f4bb6da2900 2 all 8 watchers are set, enabling cache 2026-03-20T18:05:15.706 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.707+0000 7f4bb6da2900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:05:15.707 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.707+0000 7f4bb6da2900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:05:15.707 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.707+0000 7f4bb6da2900 5 note: GC not initialized 2026-03-20T18:05:15.707 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.708+0000 7f4b66fe5640 20 reqs_thread_entry: start 2026-03-20T18:05:15.741 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.742+0000 7f4bb6da2900 20 init_complete bucket index max shards: 11 2026-03-20T18:05:15.742 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.742+0000 7f4bb6da2900 20 Filter name: none 2026-03-20T18:05:15.742 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.742+0000 7f4b64fe1640 20 reqs_thread_entry: start 2026-03-20T18:05:15.742 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.742+0000 7f4bb6da2900 20 rados->read ofs=0 len=0 2026-03-20T18:05:15.742 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.743+0000 7f4bb6da2900 20 rados_obj.operate() r=0 bl.length=267 2026-03-20T18:05:15.746 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.746+0000 7f4bb6da2900 20 remove_watcher() i=2 2026-03-20T18:05:15.746 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.747+0000 7f4bb6da2900 2 removed watcher, disabling cache 2026-03-20T18:05:15.746 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.747+0000 7f4bb6da2900 20 remove_watcher() i=0 2026-03-20T18:05:15.746 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.747+0000 7f4bb6da2900 20 remove_watcher() i=1 2026-03-20T18:05:15.746 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.747+0000 7f4bb6da2900 20 remove_watcher() i=5 2026-03-20T18:05:15.746 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.747+0000 7f4bb6da2900 20 remove_watcher() i=4 2026-03-20T18:05:15.746 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.747+0000 7f4bb6da2900 20 remove_watcher() i=3 2026-03-20T18:05:15.746 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.747+0000 7f4bb6da2900 20 remove_watcher() i=7 2026-03-20T18:05:15.746 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:15.747+0000 7f4bb6da2900 20 remove_watcher() i=6 2026-03-20T18:05:17.457 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:05:17.457 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:05:17.461 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.461+0000 7f33f3329900 -1 asok(0x55a967c835c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.67670.asok': (13) Permission denied 2026-03-20T18:05:17.478 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.478+0000 7f33f3329900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.478 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.479+0000 7f33f3329900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:17.478 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.479+0000 7f33f3329900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.479 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.480+0000 7f33f3329900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:17.479 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.480+0000 7f33f3329900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:05:17.479 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.480+0000 7f33f3329900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.479 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.480+0000 7f33f3329900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:17.479 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.480+0000 7f33f3329900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.480 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.481+0000 7f33f3329900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:05:17.480 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.481+0000 7f33f3329900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.480 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.481+0000 7f33f3329900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:05:17.481 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.481+0000 7f33f3329900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.481 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.482+0000 7f33f3329900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:17.481 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.482+0000 7f33f3329900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.481 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.482+0000 7f33f3329900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:17.481 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.482+0000 7f33f3329900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.482 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.482+0000 7f33f3329900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:17.482 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.482+0000 7f33f3329900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.483+0000 7f33f3329900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.483+0000 7f33f3329900 20 zone MyZone found 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.483+0000 7f33f3329900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.483+0000 7f33f3329900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.483+0000 7f33f3329900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.483+0000 7f33f3329900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.483+0000 7f33f3329900 20 period zonegroup name MyZoneGroup 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.483+0000 7f33f3329900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.483+0000 7f33f3329900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.484+0000 7f33f3329900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.484+0000 7f33f3329900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.484+0000 7f33f3329900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.484+0000 7f33f3329900 20 started sync module instance, tier type = 2026-03-20T18:05:17.483 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.484+0000 7f33f3329900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:05:17.489 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.490+0000 7f33f3329900 20 add_watcher() i=0 2026-03-20T18:05:17.489 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.490+0000 7f33f3329900 20 add_watcher() i=7 2026-03-20T18:05:17.489 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.490+0000 7f33f3329900 20 add_watcher() i=5 2026-03-20T18:05:17.489 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.490+0000 7f33f3329900 20 add_watcher() i=2 2026-03-20T18:05:17.489 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.490+0000 7f33f3329900 20 add_watcher() i=6 2026-03-20T18:05:17.490 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.490+0000 7f33f3329900 20 add_watcher() i=1 2026-03-20T18:05:17.490 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.491+0000 7f33f3329900 20 add_watcher() i=3 2026-03-20T18:05:17.490 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.491+0000 7f33f3329900 20 add_watcher() i=4 2026-03-20T18:05:17.490 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.491+0000 7f33f3329900 2 all 8 watchers are set, enabling cache 2026-03-20T18:05:17.492 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.493+0000 7f33f3329900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:05:17.492 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.493+0000 7f33f3329900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:05:17.492 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.493+0000 7f33f3329900 5 note: GC not initialized 2026-03-20T18:05:17.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.493+0000 7f33a27dc640 20 reqs_thread_entry: start 2026-03-20T18:05:17.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.529+0000 7f33f3329900 20 init_complete bucket index max shards: 11 2026-03-20T18:05:17.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.529+0000 7f33f3329900 20 Filter name: none 2026-03-20T18:05:17.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.529+0000 7f3353fff640 20 reqs_thread_entry: start 2026-03-20T18:05:17.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.530+0000 7f33f3329900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.529 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.530+0000 7f33f3329900 20 rados_obj.operate() r=0 bl.length=267 2026-03-20T18:05:17.533 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.534+0000 7f33f3329900 20 remove_watcher() i=3 2026-03-20T18:05:17.533 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.534+0000 7f33f3329900 2 removed watcher, disabling cache 2026-03-20T18:05:17.533 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.534+0000 7f33f3329900 20 remove_watcher() i=0 2026-03-20T18:05:17.533 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.534+0000 7f33f3329900 20 remove_watcher() i=1 2026-03-20T18:05:17.534 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.534+0000 7f33f3329900 20 remove_watcher() i=2 2026-03-20T18:05:17.534 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.534+0000 7f33f3329900 20 remove_watcher() i=7 2026-03-20T18:05:17.534 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.534+0000 7f33f3329900 20 remove_watcher() i=6 2026-03-20T18:05:17.534 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.534+0000 7f33f3329900 20 remove_watcher() i=5 2026-03-20T18:05:17.534 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.534+0000 7f33f3329900 20 remove_watcher() i=4 2026-03-20T18:05:17.821 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:05:17.821 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:05:17.826 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.827+0000 7f84e2d28900 -1 asok(0x55ed461cc5c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.67786.asok': (13) Permission denied 2026-03-20T18:05:17.842 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.843+0000 7f84e2d28900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.843 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.844+0000 7f84e2d28900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:17.843 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.844+0000 7f84e2d28900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.844 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.845+0000 7f84e2d28900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:17.844 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.845+0000 7f84e2d28900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:05:17.844 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.845+0000 7f84e2d28900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.844 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.845+0000 7f84e2d28900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:17.844 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.845+0000 7f84e2d28900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.845 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.845+0000 7f84e2d28900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:05:17.845 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.845+0000 7f84e2d28900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.845 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.846+0000 7f84e2d28900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:05:17.845 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.846+0000 7f84e2d28900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.846+0000 7f84e2d28900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:17.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.846+0000 7f84e2d28900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.847+0000 7f84e2d28900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:17.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.847+0000 7f84e2d28900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.847+0000 7f84e2d28900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:17.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.847+0000 7f84e2d28900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:05:17.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 20 zone MyZone found 2026-03-20T18:05:17.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:05:17.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:05:17.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:05:17.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:05:17.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 20 period zonegroup name MyZoneGroup 2026-03-20T18:05:17.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:05:17.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:17.848 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.848 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:17.848 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 20 started sync module instance, tier type = 2026-03-20T18:05:17.848 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.848+0000 7f84e2d28900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:05:17.850 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.851+0000 7f84e2d28900 20 add_watcher() i=3 2026-03-20T18:05:17.851 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.851+0000 7f84e2d28900 20 add_watcher() i=5 2026-03-20T18:05:17.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.852+0000 7f84e2d28900 20 add_watcher() i=2 2026-03-20T18:05:17.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.852+0000 7f84e2d28900 20 add_watcher() i=7 2026-03-20T18:05:17.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.853+0000 7f84e2d28900 20 add_watcher() i=6 2026-03-20T18:05:17.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.853+0000 7f84e2d28900 20 add_watcher() i=1 2026-03-20T18:05:17.853 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.854+0000 7f84e2d28900 20 add_watcher() i=4 2026-03-20T18:05:17.853 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.854+0000 7f84e2d28900 20 add_watcher() i=0 2026-03-20T18:05:17.853 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.854+0000 7f84e2d28900 2 all 8 watchers are set, enabling cache 2026-03-20T18:05:17.855 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.855+0000 7f84e2d28900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:05:17.855 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.855+0000 7f84e2d28900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:05:17.855 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.855+0000 7f84e2d28900 5 note: GC not initialized 2026-03-20T18:05:17.855 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.856+0000 7f84927dc640 20 reqs_thread_entry: start 2026-03-20T18:05:17.890 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.891+0000 7f84e2d28900 20 init_complete bucket index max shards: 11 2026-03-20T18:05:17.890 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.891+0000 7f84e2d28900 20 Filter name: none 2026-03-20T18:05:17.890 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.891+0000 7f8443fff640 20 reqs_thread_entry: start 2026-03-20T18:05:17.891 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.891+0000 7f84e2d28900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.891 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.891+0000 7f84e2d28900 20 rados_obj.operate() r=0 bl.length=267 2026-03-20T18:05:17.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.895+0000 7f84e2d28900 20 remove_watcher() i=0 2026-03-20T18:05:17.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.895+0000 7f84e2d28900 2 removed watcher, disabling cache 2026-03-20T18:05:17.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.895+0000 7f84e2d28900 20 remove_watcher() i=2 2026-03-20T18:05:17.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.896+0000 7f84e2d28900 20 remove_watcher() i=3 2026-03-20T18:05:17.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.896+0000 7f84e2d28900 20 remove_watcher() i=4 2026-03-20T18:05:17.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.896+0000 7f84e2d28900 20 remove_watcher() i=1 2026-03-20T18:05:17.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.896+0000 7f84e2d28900 20 remove_watcher() i=7 2026-03-20T18:05:17.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.896+0000 7f84e2d28900 20 remove_watcher() i=5 2026-03-20T18:05:17.896 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.896+0000 7f84e2d28900 20 remove_watcher() i=6 2026-03-20T18:05:17.953 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:05:17.953 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:05:17.956 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.957+0000 7f8492d2f900 -1 asok(0x555ee9f125c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.67882.asok': (13) Permission denied 2026-03-20T18:05:17.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.973+0000 7f8492d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.974+0000 7f8492d2f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:17.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.974+0000 7f8492d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.974 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.975+0000 7f8492d2f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:17.974 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.975+0000 7f8492d2f900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:05:17.974 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.975+0000 7f8492d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.974 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.975+0000 7f8492d2f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:17.974 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.975+0000 7f8492d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.975 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.976+0000 7f8492d2f900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:05:17.975 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.976+0000 7f8492d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.976 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.976+0000 7f8492d2f900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:05:17.976 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.976+0000 7f8492d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.976 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.977+0000 7f8492d2f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:17.976 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.977+0000 7f8492d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.976 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.977+0000 7f8492d2f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:17.976 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.977+0000 7f8492d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.977+0000 7f8492d2f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:17.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.977+0000 7f8492d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.978+0000 7f8492d2f900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:05:17.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.978+0000 7f8492d2f900 20 zone MyZone found 2026-03-20T18:05:17.977 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.978+0000 7f8492d2f900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:05:17.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.978+0000 7f8492d2f900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:05:17.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.978+0000 7f8492d2f900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:05:17.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.978+0000 7f8492d2f900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:05:17.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.978+0000 7f8492d2f900 20 period zonegroup name MyZoneGroup 2026-03-20T18:05:17.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.978+0000 7f8492d2f900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:05:17.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.978+0000 7f8492d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.978+0000 7f8492d2f900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:17.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.978+0000 7f8492d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:17.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.979+0000 7f8492d2f900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:17.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.979+0000 7f8492d2f900 20 started sync module instance, tier type = 2026-03-20T18:05:17.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.979+0000 7f8492d2f900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:05:17.985 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.985+0000 7f8492d2f900 20 add_watcher() i=2 2026-03-20T18:05:17.985 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.986+0000 7f8492d2f900 20 add_watcher() i=7 2026-03-20T18:05:17.985 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.986+0000 7f8492d2f900 20 add_watcher() i=3 2026-03-20T18:05:17.985 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.986+0000 7f8492d2f900 20 add_watcher() i=6 2026-03-20T18:05:17.985 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.986+0000 7f8492d2f900 20 add_watcher() i=5 2026-03-20T18:05:17.986 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.987+0000 7f8492d2f900 20 add_watcher() i=0 2026-03-20T18:05:17.987 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.987+0000 7f8492d2f900 20 add_watcher() i=4 2026-03-20T18:05:17.987 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.987+0000 7f8492d2f900 20 add_watcher() i=1 2026-03-20T18:05:17.987 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.987+0000 7f8492d2f900 2 all 8 watchers are set, enabling cache 2026-03-20T18:05:17.988 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.989+0000 7f8492d2f900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:05:17.988 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.989+0000 7f8492d2f900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:05:17.988 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.989+0000 7f8492d2f900 5 note: GC not initialized 2026-03-20T18:05:17.988 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:17.989+0000 7f84427dc640 20 reqs_thread_entry: start 2026-03-20T18:05:18.022 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.023+0000 7f8492d2f900 20 init_complete bucket index max shards: 11 2026-03-20T18:05:18.023 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.023+0000 7f8492d2f900 20 Filter name: none 2026-03-20T18:05:18.023 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.023+0000 7f83f3fff640 20 reqs_thread_entry: start 2026-03-20T18:05:18.023 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.023+0000 7f8492d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:18.023 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.024+0000 7f8492d2f900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:05:18.027 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.027+0000 7f8492d2f900 20 remove_watcher() i=3 2026-03-20T18:05:18.027 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.027+0000 7f8492d2f900 2 removed watcher, disabling cache 2026-03-20T18:05:18.027 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.027+0000 7f8492d2f900 20 remove_watcher() i=2 2026-03-20T18:05:18.027 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.028+0000 7f8492d2f900 20 remove_watcher() i=0 2026-03-20T18:05:18.027 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.028+0000 7f8492d2f900 20 remove_watcher() i=7 2026-03-20T18:05:18.027 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.028+0000 7f8492d2f900 20 remove_watcher() i=5 2026-03-20T18:05:18.027 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.028+0000 7f8492d2f900 20 remove_watcher() i=6 2026-03-20T18:05:18.028 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.028+0000 7f8492d2f900 20 remove_watcher() i=1 2026-03-20T18:05:18.028 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:18.028+0000 7f8492d2f900 20 remove_watcher() i=4 2026-03-20T18:05:23.067 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:05:23.067 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:05:23.071 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.071+0000 7fb86234f900 -1 asok(0x55840c00a5c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.67979.asok': (13) Permission denied 2026-03-20T18:05:23.089 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.089+0000 7fb86234f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:23.090 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.090+0000 7fb86234f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:23.090 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.090+0000 7fb86234f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:23.090 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.091+0000 7fb86234f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:23.090 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.091+0000 7fb86234f900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:05:23.090 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.091+0000 7fb86234f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:23.090 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.091+0000 7fb86234f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:23.090 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.091+0000 7fb86234f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:23.091 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.092+0000 7fb86234f900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:05:23.091 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.092+0000 7fb86234f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:23.092 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.093+0000 7fb86234f900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:05:23.092 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.093+0000 7fb86234f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:23.092 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.093+0000 7fb86234f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:23.092 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.093+0000 7fb86234f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:23.092 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.093+0000 7fb86234f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:23.092 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.093+0000 7fb86234f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:23.093 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.093+0000 7fb86234f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:23.093 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.093+0000 7fb86234f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.094+0000 7fb86234f900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.094+0000 7fb86234f900 20 zone MyZone found 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.094+0000 7fb86234f900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.094+0000 7fb86234f900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.094+0000 7fb86234f900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.094+0000 7fb86234f900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.094+0000 7fb86234f900 20 period zonegroup name MyZoneGroup 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.094+0000 7fb86234f900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.094+0000 7fb86234f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.095+0000 7fb86234f900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.095+0000 7fb86234f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.095+0000 7fb86234f900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.095+0000 7fb86234f900 20 started sync module instance, tier type = 2026-03-20T18:05:23.094 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.095+0000 7fb86234f900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:05:23.098 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.098+0000 7fb86234f900 20 add_watcher() i=3 2026-03-20T18:05:23.098 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.098+0000 7fb86234f900 20 add_watcher() i=5 2026-03-20T18:05:23.098 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.099+0000 7fb86234f900 20 add_watcher() i=1 2026-03-20T18:05:23.098 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.099+0000 7fb86234f900 20 add_watcher() i=6 2026-03-20T18:05:23.099 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.100+0000 7fb86234f900 20 add_watcher() i=0 2026-03-20T18:05:23.099 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.100+0000 7fb86234f900 20 add_watcher() i=2 2026-03-20T18:05:23.099 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.100+0000 7fb86234f900 20 add_watcher() i=7 2026-03-20T18:05:23.100 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.101+0000 7fb86234f900 20 add_watcher() i=4 2026-03-20T18:05:23.100 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.101+0000 7fb86234f900 2 all 8 watchers are set, enabling cache 2026-03-20T18:05:23.103 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.103+0000 7fb86234f900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:05:23.103 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.103+0000 7fb86234f900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:05:23.103 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.103+0000 7fb86234f900 5 note: GC not initialized 2026-03-20T18:05:23.103 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.104+0000 7fb810fe1640 20 reqs_thread_entry: start 2026-03-20T18:05:23.140 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.140+0000 7fb86234f900 20 init_complete bucket index max shards: 11 2026-03-20T18:05:23.140 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.141+0000 7fb86234f900 20 Filter name: none 2026-03-20T18:05:23.140 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.141+0000 7fb7c27fc640 20 reqs_thread_entry: start 2026-03-20T18:05:23.140 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.141+0000 7fb86234f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:23.141 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.141+0000 7fb86234f900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:05:23.145 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.145+0000 7fb86234f900 20 remove_watcher() i=5 2026-03-20T18:05:23.145 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.145+0000 7fb86234f900 2 removed watcher, disabling cache 2026-03-20T18:05:23.145 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.145+0000 7fb86234f900 20 remove_watcher() i=6 2026-03-20T18:05:23.145 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.146+0000 7fb86234f900 20 remove_watcher() i=2 2026-03-20T18:05:23.146 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.146+0000 7fb86234f900 20 remove_watcher() i=7 2026-03-20T18:05:23.146 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.146+0000 7fb86234f900 20 remove_watcher() i=3 2026-03-20T18:05:23.146 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.146+0000 7fb86234f900 20 remove_watcher() i=0 2026-03-20T18:05:23.146 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.146+0000 7fb86234f900 20 remove_watcher() i=4 2026-03-20T18:05:23.146 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:23.146+0000 7fb86234f900 20 remove_watcher() i=1 2026-03-20T18:05:28.186 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:05:28.186 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:05:28.191 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.191+0000 7f0a58580900 -1 asok(0x5580405ab5c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.68075.asok': (13) Permission denied 2026-03-20T18:05:28.206 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.206+0000 7f0a58580900 20 rados->read ofs=0 len=0 2026-03-20T18:05:28.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.207+0000 7f0a58580900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:28.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.207+0000 7f0a58580900 20 rados->read ofs=0 len=0 2026-03-20T18:05:28.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.208+0000 7f0a58580900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:28.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.208+0000 7f0a58580900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:05:28.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.208+0000 7f0a58580900 20 rados->read ofs=0 len=0 2026-03-20T18:05:28.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.208+0000 7f0a58580900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:28.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.208+0000 7f0a58580900 20 rados->read ofs=0 len=0 2026-03-20T18:05:28.208 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.209+0000 7f0a58580900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:05:28.208 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.209+0000 7f0a58580900 20 rados->read ofs=0 len=0 2026-03-20T18:05:28.209 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.209+0000 7f0a58580900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:05:28.209 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.209+0000 7f0a58580900 20 rados->read ofs=0 len=0 2026-03-20T18:05:28.209 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.210+0000 7f0a58580900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:28.209 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.210+0000 7f0a58580900 20 rados->read ofs=0 len=0 2026-03-20T18:05:28.209 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.210+0000 7f0a58580900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:28.209 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.210+0000 7f0a58580900 20 rados->read ofs=0 len=0 2026-03-20T18:05:28.210 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.210+0000 7f0a58580900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:28.210 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.210+0000 7f0a58580900 20 rados->read ofs=0 len=0 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.211+0000 7f0a58580900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.211+0000 7f0a58580900 20 zone MyZone found 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.211+0000 7f0a58580900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.211+0000 7f0a58580900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.211+0000 7f0a58580900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.211+0000 7f0a58580900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.211+0000 7f0a58580900 20 period zonegroup name MyZoneGroup 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.211+0000 7f0a58580900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.211+0000 7f0a58580900 20 rados->read ofs=0 len=0 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.212+0000 7f0a58580900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.212+0000 7f0a58580900 20 rados->read ofs=0 len=0 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.212+0000 7f0a58580900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.212+0000 7f0a58580900 20 started sync module instance, tier type = 2026-03-20T18:05:28.211 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.212+0000 7f0a58580900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:05:28.215 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.216+0000 7f0a58580900 20 add_watcher() i=5 2026-03-20T18:05:28.215 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.216+0000 7f0a58580900 20 add_watcher() i=2 2026-03-20T18:05:28.216 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.216+0000 7f0a58580900 20 add_watcher() i=6 2026-03-20T18:05:28.216 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.216+0000 7f0a58580900 20 add_watcher() i=3 2026-03-20T18:05:28.216 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.216+0000 7f0a58580900 20 add_watcher() i=7 2026-03-20T18:05:28.216 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.216+0000 7f0a58580900 20 add_watcher() i=1 2026-03-20T18:05:28.216 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.217+0000 7f0a58580900 20 add_watcher() i=4 2026-03-20T18:05:28.216 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.217+0000 7f0a58580900 20 add_watcher() i=0 2026-03-20T18:05:28.216 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.217+0000 7f0a58580900 2 all 8 watchers are set, enabling cache 2026-03-20T18:05:28.218 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.219+0000 7f0a58580900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:05:28.218 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.219+0000 7f0a58580900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:05:28.218 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.219+0000 7f0a58580900 5 note: GC not initialized 2026-03-20T18:05:28.218 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.219+0000 7f0a06fe5640 20 reqs_thread_entry: start 2026-03-20T18:05:28.253 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.253+0000 7f0a58580900 20 init_complete bucket index max shards: 11 2026-03-20T18:05:28.253 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.254+0000 7f0a58580900 20 Filter name: none 2026-03-20T18:05:28.253 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.254+0000 7f0a04fe1640 20 reqs_thread_entry: start 2026-03-20T18:05:28.253 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.254+0000 7f0a58580900 20 rados->read ofs=0 len=0 2026-03-20T18:05:28.254 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.254+0000 7f0a58580900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:05:28.257 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.258+0000 7f0a58580900 20 remove_watcher() i=4 2026-03-20T18:05:28.257 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.258+0000 7f0a58580900 2 removed watcher, disabling cache 2026-03-20T18:05:28.257 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.258+0000 7f0a58580900 20 remove_watcher() i=7 2026-03-20T18:05:28.257 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.258+0000 7f0a58580900 20 remove_watcher() i=0 2026-03-20T18:05:28.258 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.258+0000 7f0a58580900 20 remove_watcher() i=2 2026-03-20T18:05:28.258 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.258+0000 7f0a58580900 20 remove_watcher() i=6 2026-03-20T18:05:28.258 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.258+0000 7f0a58580900 20 remove_watcher() i=5 2026-03-20T18:05:28.258 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.258+0000 7f0a58580900 20 remove_watcher() i=3 2026-03-20T18:05:28.258 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:28.259+0000 7f0a58580900 20 remove_watcher() i=1 2026-03-20T18:05:33.297 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:05:33.298 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:05:33.302 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.302+0000 7f2a15e20900 -1 asok(0x55a18c8215c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.68172.asok': (13) Permission denied 2026-03-20T18:05:33.319 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.320+0000 7f2a15e20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:33.320 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.321+0000 7f2a15e20900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:33.320 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.321+0000 7f2a15e20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:33.320 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.321+0000 7f2a15e20900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:33.320 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.321+0000 7f2a15e20900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:05:33.320 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.321+0000 7f2a15e20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:33.320 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.321+0000 7f2a15e20900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:33.321 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.321+0000 7f2a15e20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:33.321 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.322+0000 7f2a15e20900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:05:33.321 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.322+0000 7f2a15e20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:33.322 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.323+0000 7f2a15e20900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:05:33.322 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.323+0000 7f2a15e20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:33.322 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.323+0000 7f2a15e20900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:33.322 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.323+0000 7f2a15e20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:33.322 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.323+0000 7f2a15e20900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:33.322 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.323+0000 7f2a15e20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:33.323 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.324+0000 7f2a15e20900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:33.323 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.324+0000 7f2a15e20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.324+0000 7f2a15e20900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.324+0000 7f2a15e20900 20 zone MyZone found 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.324+0000 7f2a15e20900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.324+0000 7f2a15e20900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.324+0000 7f2a15e20900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.324+0000 7f2a15e20900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.324+0000 7f2a15e20900 20 period zonegroup name MyZoneGroup 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.324+0000 7f2a15e20900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.324+0000 7f2a15e20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.325+0000 7f2a15e20900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.325+0000 7f2a15e20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.325+0000 7f2a15e20900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.325+0000 7f2a15e20900 20 started sync module instance, tier type = 2026-03-20T18:05:33.324 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.325+0000 7f2a15e20900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:05:33.328 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.328+0000 7f2a15e20900 20 add_watcher() i=6 2026-03-20T18:05:33.328 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.329+0000 7f2a15e20900 20 add_watcher() i=3 2026-03-20T18:05:33.328 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.329+0000 7f2a15e20900 20 add_watcher() i=1 2026-03-20T18:05:33.328 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.329+0000 7f2a15e20900 20 add_watcher() i=0 2026-03-20T18:05:33.329 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.329+0000 7f2a15e20900 20 add_watcher() i=5 2026-03-20T18:05:33.329 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.330+0000 7f2a15e20900 20 add_watcher() i=4 2026-03-20T18:05:33.329 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.330+0000 7f2a15e20900 20 add_watcher() i=2 2026-03-20T18:05:33.329 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.330+0000 7f2a15e20900 20 add_watcher() i=7 2026-03-20T18:05:33.329 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.330+0000 7f2a15e20900 2 all 8 watchers are set, enabling cache 2026-03-20T18:05:33.331 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.332+0000 7f2a15e20900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:05:33.331 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.332+0000 7f2a15e20900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:05:33.331 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.332+0000 7f2a15e20900 5 note: GC not initialized 2026-03-20T18:05:33.331 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.332+0000 7f29beffd640 20 reqs_thread_entry: start 2026-03-20T18:05:33.370 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.370+0000 7f2a15e20900 20 init_complete bucket index max shards: 11 2026-03-20T18:05:33.370 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.370+0000 7f2a15e20900 20 Filter name: none 2026-03-20T18:05:33.370 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.370+0000 7f29bcff9640 20 reqs_thread_entry: start 2026-03-20T18:05:33.370 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.371+0000 7f2a15e20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:33.370 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.371+0000 7f2a15e20900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:05:33.374 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.375+0000 7f2a15e20900 20 remove_watcher() i=7 2026-03-20T18:05:33.374 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.375+0000 7f2a15e20900 2 removed watcher, disabling cache 2026-03-20T18:05:33.374 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.375+0000 7f2a15e20900 20 remove_watcher() i=0 2026-03-20T18:05:33.374 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.375+0000 7f2a15e20900 20 remove_watcher() i=1 2026-03-20T18:05:33.374 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.375+0000 7f2a15e20900 20 remove_watcher() i=5 2026-03-20T18:05:33.374 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.375+0000 7f2a15e20900 20 remove_watcher() i=6 2026-03-20T18:05:33.374 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.375+0000 7f2a15e20900 20 remove_watcher() i=3 2026-03-20T18:05:33.374 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.375+0000 7f2a15e20900 20 remove_watcher() i=4 2026-03-20T18:05:33.375 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:33.375+0000 7f2a15e20900 20 remove_watcher() i=2 2026-03-20T18:05:38.415 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:05:38.415 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:05:38.419 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.419+0000 7efe32c0f900 -1 asok(0x5599da4f95c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.68268.asok': (13) Permission denied 2026-03-20T18:05:38.436 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.436+0000 7efe32c0f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:38.437 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.437+0000 7efe32c0f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:38.437 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.437+0000 7efe32c0f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:38.437 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.438+0000 7efe32c0f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:38.437 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.438+0000 7efe32c0f900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:05:38.437 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.438+0000 7efe32c0f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:38.437 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.438+0000 7efe32c0f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:38.437 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.438+0000 7efe32c0f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:38.438 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.439+0000 7efe32c0f900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:05:38.438 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.439+0000 7efe32c0f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:38.439 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.440+0000 7efe32c0f900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:05:38.439 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.440+0000 7efe32c0f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:38.439 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.440+0000 7efe32c0f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:38.439 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.440+0000 7efe32c0f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:38.439 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.440+0000 7efe32c0f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:38.439 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.440+0000 7efe32c0f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:38.440 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.440+0000 7efe32c0f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:38.440 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.440+0000 7efe32c0f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:38.440 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.441+0000 7efe32c0f900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:05:38.440 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.441+0000 7efe32c0f900 20 zone MyZone found 2026-03-20T18:05:38.440 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.441+0000 7efe32c0f900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:05:38.440 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.441+0000 7efe32c0f900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:05:38.440 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.441+0000 7efe32c0f900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:05:38.440 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.441+0000 7efe32c0f900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:05:38.440 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.441+0000 7efe32c0f900 20 period zonegroup name MyZoneGroup 2026-03-20T18:05:38.440 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.441+0000 7efe32c0f900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:05:38.441 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.441+0000 7efe32c0f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:38.441 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.441+0000 7efe32c0f900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:38.441 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.441+0000 7efe32c0f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:38.441 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.442+0000 7efe32c0f900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:38.441 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.442+0000 7efe32c0f900 20 started sync module instance, tier type = 2026-03-20T18:05:38.441 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.442+0000 7efe32c0f900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:05:38.445 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.445+0000 7efe32c0f900 20 add_watcher() i=3 2026-03-20T18:05:38.445 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.445+0000 7efe32c0f900 20 add_watcher() i=5 2026-03-20T18:05:38.445 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.445+0000 7efe32c0f900 20 add_watcher() i=1 2026-03-20T18:05:38.445 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.445+0000 7efe32c0f900 20 add_watcher() i=6 2026-03-20T18:05:38.446 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.446+0000 7efe32c0f900 20 add_watcher() i=7 2026-03-20T18:05:38.446 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.447+0000 7efe32c0f900 20 add_watcher() i=0 2026-03-20T18:05:38.446 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.447+0000 7efe32c0f900 20 add_watcher() i=4 2026-03-20T18:05:38.446 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.447+0000 7efe32c0f900 20 add_watcher() i=2 2026-03-20T18:05:38.446 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.447+0000 7efe32c0f900 2 all 8 watchers are set, enabling cache 2026-03-20T18:05:38.448 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.449+0000 7efe32c0f900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:05:38.448 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.449+0000 7efe32c0f900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:05:38.448 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.449+0000 7efe32c0f900 5 note: GC not initialized 2026-03-20T18:05:38.448 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.449+0000 7efde0fe1640 20 reqs_thread_entry: start 2026-03-20T18:05:38.487 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.487+0000 7efe32c0f900 20 init_complete bucket index max shards: 11 2026-03-20T18:05:38.487 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.488+0000 7efe32c0f900 20 Filter name: none 2026-03-20T18:05:38.487 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.488+0000 7efd927fc640 20 reqs_thread_entry: start 2026-03-20T18:05:38.487 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.488+0000 7efe32c0f900 20 rados->read ofs=0 len=0 2026-03-20T18:05:38.488 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.488+0000 7efe32c0f900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:05:38.492 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.493+0000 7efe32c0f900 20 remove_watcher() i=0 2026-03-20T18:05:38.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.493+0000 7efe32c0f900 2 removed watcher, disabling cache 2026-03-20T18:05:38.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.493+0000 7efe32c0f900 20 remove_watcher() i=4 2026-03-20T18:05:38.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.493+0000 7efe32c0f900 20 remove_watcher() i=1 2026-03-20T18:05:38.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.493+0000 7efe32c0f900 20 remove_watcher() i=3 2026-03-20T18:05:38.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.493+0000 7efe32c0f900 20 remove_watcher() i=7 2026-03-20T18:05:38.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.493+0000 7efe32c0f900 20 remove_watcher() i=2 2026-03-20T18:05:38.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.493+0000 7efe32c0f900 20 remove_watcher() i=5 2026-03-20T18:05:38.493 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:38.493+0000 7efe32c0f900 20 remove_watcher() i=6 2026-03-20T18:05:43.529 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:05:43.529 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:05:43.534 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.535+0000 7f400473b900 -1 asok(0x563624c585c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.68365.asok': (13) Permission denied 2026-03-20T18:05:43.551 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.552+0000 7f400473b900 20 rados->read ofs=0 len=0 2026-03-20T18:05:43.552 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.553+0000 7f400473b900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:43.553 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.553+0000 7f400473b900 20 rados->read ofs=0 len=0 2026-03-20T18:05:43.553 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.554+0000 7f400473b900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:43.553 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.554+0000 7f400473b900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:05:43.553 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.554+0000 7f400473b900 20 rados->read ofs=0 len=0 2026-03-20T18:05:43.553 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.554+0000 7f400473b900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:43.553 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.554+0000 7f400473b900 20 rados->read ofs=0 len=0 2026-03-20T18:05:43.554 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.555+0000 7f400473b900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:05:43.554 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.555+0000 7f400473b900 20 rados->read ofs=0 len=0 2026-03-20T18:05:43.555 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.555+0000 7f400473b900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:05:43.555 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.555+0000 7f400473b900 20 rados->read ofs=0 len=0 2026-03-20T18:05:43.555 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.556+0000 7f400473b900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:43.555 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.556+0000 7f400473b900 20 rados->read ofs=0 len=0 2026-03-20T18:05:43.555 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.556+0000 7f400473b900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:43.555 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.556+0000 7f400473b900 20 rados->read ofs=0 len=0 2026-03-20T18:05:43.556 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.556+0000 7f400473b900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:43.556 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.556+0000 7f400473b900 20 rados->read ofs=0 len=0 2026-03-20T18:05:43.556 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.557+0000 7f400473b900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:05:43.556 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.557+0000 7f400473b900 20 zone MyZone found 2026-03-20T18:05:43.556 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.557+0000 7f400473b900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:05:43.556 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.557+0000 7f400473b900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:05:43.556 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.557+0000 7f400473b900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:05:43.556 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.557+0000 7f400473b900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:05:43.556 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.557+0000 7f400473b900 20 period zonegroup name MyZoneGroup 2026-03-20T18:05:43.556 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.557+0000 7f400473b900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:05:43.556 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.557+0000 7f400473b900 20 rados->read ofs=0 len=0 2026-03-20T18:05:43.557 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.557+0000 7f400473b900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:43.557 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.557+0000 7f400473b900 20 rados->read ofs=0 len=0 2026-03-20T18:05:43.557 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.557+0000 7f400473b900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:43.557 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.558+0000 7f400473b900 20 started sync module instance, tier type = 2026-03-20T18:05:43.557 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.558+0000 7f400473b900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:05:43.561 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.561+0000 7f400473b900 20 add_watcher() i=3 2026-03-20T18:05:43.561 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.562+0000 7f400473b900 20 add_watcher() i=6 2026-03-20T18:05:43.561 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.562+0000 7f400473b900 20 add_watcher() i=7 2026-03-20T18:05:43.562 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.562+0000 7f400473b900 20 add_watcher() i=2 2026-03-20T18:05:43.562 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.562+0000 7f400473b900 20 add_watcher() i=5 2026-03-20T18:05:43.562 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.563+0000 7f400473b900 20 add_watcher() i=4 2026-03-20T18:05:43.563 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.564+0000 7f400473b900 20 add_watcher() i=0 2026-03-20T18:05:43.564 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.565+0000 7f400473b900 20 add_watcher() i=1 2026-03-20T18:05:43.564 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.565+0000 7f400473b900 2 all 8 watchers are set, enabling cache 2026-03-20T18:05:43.566 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.567+0000 7f400473b900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:05:43.566 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.567+0000 7f400473b900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:05:43.566 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.567+0000 7f400473b900 5 note: GC not initialized 2026-03-20T18:05:43.566 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.567+0000 7f3fb2fe5640 20 reqs_thread_entry: start 2026-03-20T18:05:43.608 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.609+0000 7f400473b900 20 init_complete bucket index max shards: 11 2026-03-20T18:05:43.608 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.609+0000 7f400473b900 20 Filter name: none 2026-03-20T18:05:43.608 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.609+0000 7f3fb0fe1640 20 reqs_thread_entry: start 2026-03-20T18:05:43.609 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.609+0000 7f400473b900 20 rados->read ofs=0 len=0 2026-03-20T18:05:43.609 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.610+0000 7f400473b900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:05:43.614 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.614+0000 7f400473b900 20 remove_watcher() i=5 2026-03-20T18:05:43.614 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.614+0000 7f400473b900 2 removed watcher, disabling cache 2026-03-20T18:05:43.614 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.614+0000 7f400473b900 20 remove_watcher() i=2 2026-03-20T18:05:43.614 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.614+0000 7f400473b900 20 remove_watcher() i=6 2026-03-20T18:05:43.614 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.614+0000 7f400473b900 20 remove_watcher() i=7 2026-03-20T18:05:43.614 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.615+0000 7f400473b900 20 remove_watcher() i=1 2026-03-20T18:05:43.615 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.615+0000 7f400473b900 20 remove_watcher() i=3 2026-03-20T18:05:43.615 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.616+0000 7f400473b900 20 remove_watcher() i=4 2026-03-20T18:05:43.615 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:43.616+0000 7f400473b900 20 remove_watcher() i=0 2026-03-20T18:05:48.748 INFO:teuthology.orchestra.run.vm01.stderr:ok 2026-03-20T18:05:57.758 INFO:teuthology.orchestra.run.vm01.stderr:test pushing persistent notification amqp ... ignoring --setuser ceph since I am not root 2026-03-20T18:05:57.758 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:05:57.761 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.762+0000 7f5cb9a20900 -1 asok(0x5577d64f15c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.68566.asok': (13) Permission denied 2026-03-20T18:05:57.777 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.778+0000 7f5cb9a20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:57.778 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.778+0000 7f5cb9a20900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:57.778 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.778+0000 7f5cb9a20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:57.778 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.779+0000 7f5cb9a20900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:57.778 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.779+0000 7f5cb9a20900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:05:57.778 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.779+0000 7f5cb9a20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:57.778 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.779+0000 7f5cb9a20900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:57.778 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.779+0000 7f5cb9a20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:57.779 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.780+0000 7f5cb9a20900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:05:57.779 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.780+0000 7f5cb9a20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:57.780 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.781+0000 7f5cb9a20900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:05:57.780 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.781+0000 7f5cb9a20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:57.780 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.781+0000 7f5cb9a20900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:57.780 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.781+0000 7f5cb9a20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:57.780 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.781+0000 7f5cb9a20900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:05:57.780 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.781+0000 7f5cb9a20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:57.781 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.782+0000 7f5cb9a20900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:05:57.781 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.782+0000 7f5cb9a20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.782+0000 7f5cb9a20900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.782+0000 7f5cb9a20900 20 zone MyZone found 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.782+0000 7f5cb9a20900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.782+0000 7f5cb9a20900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.782+0000 7f5cb9a20900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.782+0000 7f5cb9a20900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.782+0000 7f5cb9a20900 20 period zonegroup name MyZoneGroup 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.782+0000 7f5cb9a20900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.782+0000 7f5cb9a20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.782+0000 7f5cb9a20900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.783+0000 7f5cb9a20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.783+0000 7f5cb9a20900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.783+0000 7f5cb9a20900 20 started sync module instance, tier type = 2026-03-20T18:05:57.782 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.783+0000 7f5cb9a20900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:05:57.786 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.786+0000 7f5cb9a20900 20 add_watcher() i=3 2026-03-20T18:05:57.786 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.786+0000 7f5cb9a20900 20 add_watcher() i=5 2026-03-20T18:05:57.786 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.787+0000 7f5cb9a20900 20 add_watcher() i=2 2026-03-20T18:05:57.786 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.787+0000 7f5cb9a20900 20 add_watcher() i=0 2026-03-20T18:05:57.786 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.787+0000 7f5cb9a20900 20 add_watcher() i=1 2026-03-20T18:05:57.786 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.787+0000 7f5cb9a20900 20 add_watcher() i=6 2026-03-20T18:05:57.787 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.788+0000 7f5cb9a20900 20 add_watcher() i=4 2026-03-20T18:05:57.787 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.788+0000 7f5cb9a20900 20 add_watcher() i=7 2026-03-20T18:05:57.787 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.788+0000 7f5cb9a20900 2 all 8 watchers are set, enabling cache 2026-03-20T18:05:57.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.790+0000 7f5cb9a20900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:05:57.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.790+0000 7f5cb9a20900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:05:57.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.790+0000 7f5cb9a20900 5 note: GC not initialized 2026-03-20T18:05:57.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.790+0000 7f5c62ffd640 20 reqs_thread_entry: start 2026-03-20T18:05:57.827 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.828+0000 7f5cb9a20900 20 init_complete bucket index max shards: 11 2026-03-20T18:05:57.827 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.828+0000 7f5cb9a20900 20 Filter name: none 2026-03-20T18:05:57.827 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.828+0000 7f5c60ff9640 20 reqs_thread_entry: start 2026-03-20T18:05:57.827 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.828+0000 7f5cb9a20900 20 rados->read ofs=0 len=0 2026-03-20T18:05:57.828 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.828+0000 7f5cb9a20900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:05:57.831 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.832+0000 7f5cb9a20900 20 remove_watcher() i=7 2026-03-20T18:05:57.832 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.832+0000 7f5cb9a20900 2 removed watcher, disabling cache 2026-03-20T18:05:57.832 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.832+0000 7f5cb9a20900 20 remove_watcher() i=0 2026-03-20T18:05:57.832 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.833+0000 7f5cb9a20900 20 remove_watcher() i=4 2026-03-20T18:05:57.832 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.833+0000 7f5cb9a20900 20 remove_watcher() i=1 2026-03-20T18:05:57.832 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.833+0000 7f5cb9a20900 20 remove_watcher() i=6 2026-03-20T18:05:57.832 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.833+0000 7f5cb9a20900 20 remove_watcher() i=5 2026-03-20T18:05:57.833 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.833+0000 7f5cb9a20900 20 remove_watcher() i=3 2026-03-20T18:05:57.833 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:05:57.833+0000 7f5cb9a20900 20 remove_watcher() i=2 2026-03-20T18:06:02.874 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:06:02.874 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:06:02.878 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.879+0000 7f8c37f2a900 -1 asok(0x56014fc5e5c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.68665.asok': (13) Permission denied 2026-03-20T18:06:02.894 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.894+0000 7f8c37f2a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:02.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.896+0000 7f8c37f2a900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:02.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.896+0000 7f8c37f2a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:02.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.896+0000 7f8c37f2a900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:02.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.896+0000 7f8c37f2a900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:06:02.895 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.896+0000 7f8c37f2a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:02.896 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.896+0000 7f8c37f2a900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:02.896 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.896+0000 7f8c37f2a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:02.896 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.897+0000 7f8c37f2a900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:06:02.896 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.897+0000 7f8c37f2a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:02.897 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.898+0000 7f8c37f2a900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:06:02.897 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.898+0000 7f8c37f2a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:02.897 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.898+0000 7f8c37f2a900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:02.897 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.898+0000 7f8c37f2a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:02.897 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.898+0000 7f8c37f2a900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:02.897 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.898+0000 7f8c37f2a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:02.898 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.898+0000 7f8c37f2a900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:02.898 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.898+0000 7f8c37f2a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:02.898 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.899+0000 7f8c37f2a900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.899+0000 7f8c37f2a900 20 zone MyZone found 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.899+0000 7f8c37f2a900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.899+0000 7f8c37f2a900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.899+0000 7f8c37f2a900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.899+0000 7f8c37f2a900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.899+0000 7f8c37f2a900 20 period zonegroup name MyZoneGroup 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.899+0000 7f8c37f2a900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.899+0000 7f8c37f2a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.899+0000 7f8c37f2a900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.899+0000 7f8c37f2a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.900+0000 7f8c37f2a900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.900+0000 7f8c37f2a900 20 started sync module instance, tier type = 2026-03-20T18:06:02.899 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.900+0000 7f8c37f2a900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:06:02.903 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.903+0000 7f8c37f2a900 20 add_watcher() i=1 2026-03-20T18:06:02.903 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.904+0000 7f8c37f2a900 20 add_watcher() i=3 2026-03-20T18:06:02.903 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.904+0000 7f8c37f2a900 20 add_watcher() i=5 2026-03-20T18:06:02.903 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.904+0000 7f8c37f2a900 20 add_watcher() i=7 2026-03-20T18:06:02.903 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.904+0000 7f8c37f2a900 20 add_watcher() i=6 2026-03-20T18:06:02.904 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.904+0000 7f8c37f2a900 20 add_watcher() i=0 2026-03-20T18:06:02.904 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.905+0000 7f8c37f2a900 20 add_watcher() i=4 2026-03-20T18:06:02.904 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.905+0000 7f8c37f2a900 20 add_watcher() i=2 2026-03-20T18:06:02.904 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.905+0000 7f8c37f2a900 2 all 8 watchers are set, enabling cache 2026-03-20T18:06:02.906 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.907+0000 7f8c37f2a900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:06:02.906 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.907+0000 7f8c37f2a900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:06:02.906 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.907+0000 7f8c37f2a900 5 note: GC not initialized 2026-03-20T18:06:02.906 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.907+0000 7f8be5fe3640 20 reqs_thread_entry: start 2026-03-20T18:06:02.944 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.944+0000 7f8c37f2a900 20 init_complete bucket index max shards: 11 2026-03-20T18:06:02.944 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.944+0000 7f8c37f2a900 20 Filter name: none 2026-03-20T18:06:02.944 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.944+0000 7f8b977fe640 20 reqs_thread_entry: start 2026-03-20T18:06:02.944 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.945+0000 7f8c37f2a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:02.944 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.945+0000 7f8c37f2a900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:06:02.948 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.949+0000 7f8c37f2a900 20 remove_watcher() i=5 2026-03-20T18:06:02.948 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.949+0000 7f8c37f2a900 2 removed watcher, disabling cache 2026-03-20T18:06:02.948 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.949+0000 7f8c37f2a900 20 remove_watcher() i=3 2026-03-20T18:06:02.948 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.949+0000 7f8c37f2a900 20 remove_watcher() i=0 2026-03-20T18:06:02.949 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.949+0000 7f8c37f2a900 20 remove_watcher() i=7 2026-03-20T18:06:02.949 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.950+0000 7f8c37f2a900 20 remove_watcher() i=4 2026-03-20T18:06:02.949 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.950+0000 7f8c37f2a900 20 remove_watcher() i=6 2026-03-20T18:06:02.949 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.950+0000 7f8c37f2a900 20 remove_watcher() i=1 2026-03-20T18:06:02.949 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:02.950+0000 7f8c37f2a900 20 remove_watcher() i=2 2026-03-20T18:06:07.997 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:06:07.997 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:06:08.000 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.001+0000 7f3dc874b900 -1 asok(0x557bc91e05c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.68761.asok': (13) Permission denied 2026-03-20T18:06:08.016 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.016+0000 7f3dc874b900 20 rados->read ofs=0 len=0 2026-03-20T18:06:08.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.017+0000 7f3dc874b900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:08.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.017+0000 7f3dc874b900 20 rados->read ofs=0 len=0 2026-03-20T18:06:08.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.018+0000 7f3dc874b900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:08.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.018+0000 7f3dc874b900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:06:08.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.018+0000 7f3dc874b900 20 rados->read ofs=0 len=0 2026-03-20T18:06:08.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.018+0000 7f3dc874b900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:08.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.018+0000 7f3dc874b900 20 rados->read ofs=0 len=0 2026-03-20T18:06:08.018 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.019+0000 7f3dc874b900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:06:08.018 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.019+0000 7f3dc874b900 20 rados->read ofs=0 len=0 2026-03-20T18:06:08.019 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.020+0000 7f3dc874b900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:06:08.019 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.020+0000 7f3dc874b900 20 rados->read ofs=0 len=0 2026-03-20T18:06:08.019 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.020+0000 7f3dc874b900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:08.019 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.020+0000 7f3dc874b900 20 rados->read ofs=0 len=0 2026-03-20T18:06:08.019 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.020+0000 7f3dc874b900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:08.019 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.020+0000 7f3dc874b900 20 rados->read ofs=0 len=0 2026-03-20T18:06:08.020 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.021+0000 7f3dc874b900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:08.020 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.021+0000 7f3dc874b900 20 rados->read ofs=0 len=0 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.021+0000 7f3dc874b900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.021+0000 7f3dc874b900 20 zone MyZone found 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.021+0000 7f3dc874b900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.021+0000 7f3dc874b900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.021+0000 7f3dc874b900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.021+0000 7f3dc874b900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.021+0000 7f3dc874b900 20 period zonegroup name MyZoneGroup 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.021+0000 7f3dc874b900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.021+0000 7f3dc874b900 20 rados->read ofs=0 len=0 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.022+0000 7f3dc874b900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.022+0000 7f3dc874b900 20 rados->read ofs=0 len=0 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.022+0000 7f3dc874b900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.022+0000 7f3dc874b900 20 started sync module instance, tier type = 2026-03-20T18:06:08.021 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.022+0000 7f3dc874b900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:06:08.025 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.026+0000 7f3dc874b900 20 add_watcher() i=7 2026-03-20T18:06:08.025 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.026+0000 7f3dc874b900 20 add_watcher() i=0 2026-03-20T18:06:08.025 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.026+0000 7f3dc874b900 20 add_watcher() i=1 2026-03-20T18:06:08.026 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.026+0000 7f3dc874b900 20 add_watcher() i=4 2026-03-20T18:06:08.026 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.026+0000 7f3dc874b900 20 add_watcher() i=6 2026-03-20T18:06:08.026 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.027+0000 7f3dc874b900 20 add_watcher() i=2 2026-03-20T18:06:08.026 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.027+0000 7f3dc874b900 20 add_watcher() i=5 2026-03-20T18:06:08.026 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.027+0000 7f3dc874b900 20 add_watcher() i=3 2026-03-20T18:06:08.026 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.027+0000 7f3dc874b900 2 all 8 watchers are set, enabling cache 2026-03-20T18:06:08.028 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.029+0000 7f3dc874b900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:06:08.028 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.029+0000 7f3dc874b900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:06:08.028 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.029+0000 7f3dc874b900 5 note: GC not initialized 2026-03-20T18:06:08.028 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.029+0000 7f3d78fe1640 20 reqs_thread_entry: start 2026-03-20T18:06:08.066 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.066+0000 7f3dc874b900 20 init_complete bucket index max shards: 11 2026-03-20T18:06:08.066 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.066+0000 7f3dc874b900 20 Filter name: none 2026-03-20T18:06:08.066 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.067+0000 7f3d2a7fc640 20 reqs_thread_entry: start 2026-03-20T18:06:08.066 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.067+0000 7f3dc874b900 20 rados->read ofs=0 len=0 2026-03-20T18:06:08.067 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.067+0000 7f3dc874b900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:06:08.072 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.073+0000 7f3dc874b900 20 remove_watcher() i=0 2026-03-20T18:06:08.072 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.073+0000 7f3dc874b900 2 removed watcher, disabling cache 2026-03-20T18:06:08.073 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.073+0000 7f3dc874b900 20 remove_watcher() i=2 2026-03-20T18:06:08.073 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.073+0000 7f3dc874b900 20 remove_watcher() i=3 2026-03-20T18:06:08.073 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.073+0000 7f3dc874b900 20 remove_watcher() i=4 2026-03-20T18:06:08.073 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.073+0000 7f3dc874b900 20 remove_watcher() i=1 2026-03-20T18:06:08.073 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.073+0000 7f3dc874b900 20 remove_watcher() i=5 2026-03-20T18:06:08.073 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.073+0000 7f3dc874b900 20 remove_watcher() i=6 2026-03-20T18:06:08.073 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:08.074+0000 7f3dc874b900 20 remove_watcher() i=7 2026-03-20T18:06:13.122 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:06:13.122 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:06:13.126 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.127+0000 7ff523b23900 -1 asok(0x5599013265c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.68860.asok': (13) Permission denied 2026-03-20T18:06:13.143 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.144+0000 7ff523b23900 20 rados->read ofs=0 len=0 2026-03-20T18:06:13.144 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.145+0000 7ff523b23900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:13.144 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.145+0000 7ff523b23900 20 rados->read ofs=0 len=0 2026-03-20T18:06:13.145 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.145+0000 7ff523b23900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:13.145 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.145+0000 7ff523b23900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:06:13.145 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.145+0000 7ff523b23900 20 rados->read ofs=0 len=0 2026-03-20T18:06:13.145 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.145+0000 7ff523b23900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:13.145 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.145+0000 7ff523b23900 20 rados->read ofs=0 len=0 2026-03-20T18:06:13.146 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.146+0000 7ff523b23900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:06:13.146 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.146+0000 7ff523b23900 20 rados->read ofs=0 len=0 2026-03-20T18:06:13.146 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.147+0000 7ff523b23900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:06:13.146 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.147+0000 7ff523b23900 20 rados->read ofs=0 len=0 2026-03-20T18:06:13.147 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.147+0000 7ff523b23900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:13.147 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.147+0000 7ff523b23900 20 rados->read ofs=0 len=0 2026-03-20T18:06:13.147 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.148+0000 7ff523b23900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:13.147 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.148+0000 7ff523b23900 20 rados->read ofs=0 len=0 2026-03-20T18:06:13.147 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.148+0000 7ff523b23900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:13.147 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.148+0000 7ff523b23900 20 rados->read ofs=0 len=0 2026-03-20T18:06:13.148 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:06:13.148 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 20 zone MyZone found 2026-03-20T18:06:13.148 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:06:13.148 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:06:13.148 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:06:13.148 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:06:13.148 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 20 period zonegroup name MyZoneGroup 2026-03-20T18:06:13.148 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:06:13.148 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 20 rados->read ofs=0 len=0 2026-03-20T18:06:13.149 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:06:13.149 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 20 rados->read ofs=0 len=0 2026-03-20T18:06:13.149 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:06:13.149 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 20 started sync module instance, tier type = 2026-03-20T18:06:13.149 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.149+0000 7ff523b23900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:06:13.153 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.153+0000 7ff523b23900 20 add_watcher() i=3 2026-03-20T18:06:13.153 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.154+0000 7ff523b23900 20 add_watcher() i=2 2026-03-20T18:06:13.153 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.154+0000 7ff523b23900 20 add_watcher() i=5 2026-03-20T18:06:13.153 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.154+0000 7ff523b23900 20 add_watcher() i=6 2026-03-20T18:06:13.153 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.154+0000 7ff523b23900 20 add_watcher() i=7 2026-03-20T18:06:13.154 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.155+0000 7ff523b23900 20 add_watcher() i=1 2026-03-20T18:06:13.156 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.157+0000 7ff523b23900 20 add_watcher() i=0 2026-03-20T18:06:13.157 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.157+0000 7ff523b23900 20 add_watcher() i=4 2026-03-20T18:06:13.157 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.157+0000 7ff523b23900 2 all 8 watchers are set, enabling cache 2026-03-20T18:06:13.159 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.160+0000 7ff523b23900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:06:13.159 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.160+0000 7ff523b23900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:06:13.159 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.160+0000 7ff523b23900 5 note: GC not initialized 2026-03-20T18:06:13.159 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.160+0000 7ff4d27dc640 20 reqs_thread_entry: start 2026-03-20T18:06:13.200 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.201+0000 7ff523b23900 20 init_complete bucket index max shards: 11 2026-03-20T18:06:13.200 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.201+0000 7ff523b23900 20 Filter name: none 2026-03-20T18:06:13.200 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.201+0000 7ff483fff640 20 reqs_thread_entry: start 2026-03-20T18:06:13.201 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.201+0000 7ff523b23900 20 rados->read ofs=0 len=0 2026-03-20T18:06:13.201 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.202+0000 7ff523b23900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:06:13.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.207+0000 7ff523b23900 20 remove_watcher() i=4 2026-03-20T18:06:13.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.207+0000 7ff523b23900 2 removed watcher, disabling cache 2026-03-20T18:06:13.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.207+0000 7ff523b23900 20 remove_watcher() i=0 2026-03-20T18:06:13.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.207+0000 7ff523b23900 20 remove_watcher() i=1 2026-03-20T18:06:13.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.207+0000 7ff523b23900 20 remove_watcher() i=3 2026-03-20T18:06:13.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.208+0000 7ff523b23900 20 remove_watcher() i=7 2026-03-20T18:06:13.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.208+0000 7ff523b23900 20 remove_watcher() i=5 2026-03-20T18:06:13.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.208+0000 7ff523b23900 20 remove_watcher() i=2 2026-03-20T18:06:13.207 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:13.208+0000 7ff523b23900 20 remove_watcher() i=6 2026-03-20T18:06:18.255 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:06:18.255 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:06:18.259 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.260+0000 7f64a2210900 -1 asok(0x559595c1e5c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.68958.asok': (13) Permission denied 2026-03-20T18:06:18.277 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.278+0000 7f64a2210900 20 rados->read ofs=0 len=0 2026-03-20T18:06:18.278 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.279+0000 7f64a2210900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:18.278 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.279+0000 7f64a2210900 20 rados->read ofs=0 len=0 2026-03-20T18:06:18.279 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.280+0000 7f64a2210900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:18.279 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.280+0000 7f64a2210900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:06:18.279 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.280+0000 7f64a2210900 20 rados->read ofs=0 len=0 2026-03-20T18:06:18.279 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.280+0000 7f64a2210900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:18.279 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.280+0000 7f64a2210900 20 rados->read ofs=0 len=0 2026-03-20T18:06:18.280 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.281+0000 7f64a2210900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:06:18.280 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.281+0000 7f64a2210900 20 rados->read ofs=0 len=0 2026-03-20T18:06:18.281 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.281+0000 7f64a2210900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:06:18.281 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.281+0000 7f64a2210900 20 rados->read ofs=0 len=0 2026-03-20T18:06:18.281 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.282+0000 7f64a2210900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:18.281 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.282+0000 7f64a2210900 20 rados->read ofs=0 len=0 2026-03-20T18:06:18.281 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.282+0000 7f64a2210900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:18.281 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.282+0000 7f64a2210900 20 rados->read ofs=0 len=0 2026-03-20T18:06:18.282 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.283+0000 7f64a2210900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:18.282 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.283+0000 7f64a2210900 20 rados->read ofs=0 len=0 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.283+0000 7f64a2210900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.283+0000 7f64a2210900 20 zone MyZone found 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.283+0000 7f64a2210900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.283+0000 7f64a2210900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.283+0000 7f64a2210900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.283+0000 7f64a2210900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.283+0000 7f64a2210900 20 period zonegroup name MyZoneGroup 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.283+0000 7f64a2210900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.283+0000 7f64a2210900 20 rados->read ofs=0 len=0 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.284+0000 7f64a2210900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.284+0000 7f64a2210900 20 rados->read ofs=0 len=0 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.284+0000 7f64a2210900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.284+0000 7f64a2210900 20 started sync module instance, tier type = 2026-03-20T18:06:18.283 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.284+0000 7f64a2210900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:06:18.287 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.287+0000 7f64a2210900 20 add_watcher() i=3 2026-03-20T18:06:18.288 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.288+0000 7f64a2210900 20 add_watcher() i=1 2026-03-20T18:06:18.288 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.288+0000 7f64a2210900 20 add_watcher() i=4 2026-03-20T18:06:18.288 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.288+0000 7f64a2210900 20 add_watcher() i=7 2026-03-20T18:06:18.288 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.289+0000 7f64a2210900 20 add_watcher() i=0 2026-03-20T18:06:18.289 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.289+0000 7f64a2210900 20 add_watcher() i=5 2026-03-20T18:06:18.289 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.289+0000 7f64a2210900 20 add_watcher() i=6 2026-03-20T18:06:18.289 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.290+0000 7f64a2210900 20 add_watcher() i=2 2026-03-20T18:06:18.289 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.290+0000 7f64a2210900 2 all 8 watchers are set, enabling cache 2026-03-20T18:06:18.291 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.292+0000 7f64a2210900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:06:18.291 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.292+0000 7f64a2210900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:06:18.291 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.292+0000 7f64a2210900 5 note: GC not initialized 2026-03-20T18:06:18.291 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.292+0000 7f6403fff640 20 reqs_thread_entry: start 2026-03-20T18:06:18.334 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.334+0000 7f64a2210900 20 init_complete bucket index max shards: 11 2026-03-20T18:06:18.334 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.334+0000 7f64a2210900 20 Filter name: none 2026-03-20T18:06:18.334 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.334+0000 7f6401ffb640 20 reqs_thread_entry: start 2026-03-20T18:06:18.334 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.335+0000 7f64a2210900 20 rados->read ofs=0 len=0 2026-03-20T18:06:18.334 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.335+0000 7f64a2210900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:06:18.338 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.339+0000 7f64a2210900 20 remove_watcher() i=0 2026-03-20T18:06:18.339 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.339+0000 7f64a2210900 2 removed watcher, disabling cache 2026-03-20T18:06:18.339 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.339+0000 7f64a2210900 20 remove_watcher() i=3 2026-03-20T18:06:18.339 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.340+0000 7f64a2210900 20 remove_watcher() i=5 2026-03-20T18:06:18.339 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.340+0000 7f64a2210900 20 remove_watcher() i=4 2026-03-20T18:06:18.339 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.340+0000 7f64a2210900 20 remove_watcher() i=7 2026-03-20T18:06:18.339 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.340+0000 7f64a2210900 20 remove_watcher() i=1 2026-03-20T18:06:18.339 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.340+0000 7f64a2210900 20 remove_watcher() i=6 2026-03-20T18:06:18.339 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:18.340+0000 7f64a2210900 20 remove_watcher() i=2 2026-03-20T18:06:24.619 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:06:24.619 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:06:24.623 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.623+0000 7ff72252a900 -1 asok(0x55de86b295c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.69155.asok': (13) Permission denied 2026-03-20T18:06:24.640 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.641+0000 7ff72252a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:24.642 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.642+0000 7ff72252a900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:24.642 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.642+0000 7ff72252a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:24.643 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.643+0000 7ff72252a900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:24.643 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.643+0000 7ff72252a900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:06:24.643 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.643+0000 7ff72252a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:24.643 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.643+0000 7ff72252a900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:24.643 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.643+0000 7ff72252a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:24.644 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.644+0000 7ff72252a900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:06:24.644 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.644+0000 7ff72252a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:24.644 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.645+0000 7ff72252a900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:06:24.644 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.645+0000 7ff72252a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:24.644 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.645+0000 7ff72252a900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:24.644 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.645+0000 7ff72252a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:24.645 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.645+0000 7ff72252a900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:24.645 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.645+0000 7ff72252a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:24.645 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.646+0000 7ff72252a900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:24.645 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.646+0000 7ff72252a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 20 zone MyZone found 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 20 period zonegroup name MyZoneGroup 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 20 started sync module instance, tier type = 2026-03-20T18:06:24.647 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.647+0000 7ff72252a900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:06:24.651 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.652+0000 7ff72252a900 20 add_watcher() i=3 2026-03-20T18:06:24.651 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.652+0000 7ff72252a900 20 add_watcher() i=0 2026-03-20T18:06:24.651 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.652+0000 7ff72252a900 20 add_watcher() i=1 2026-03-20T18:06:24.651 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.652+0000 7ff72252a900 20 add_watcher() i=4 2026-03-20T18:06:24.652 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.652+0000 7ff72252a900 20 add_watcher() i=7 2026-03-20T18:06:24.652 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.652+0000 7ff72252a900 20 add_watcher() i=5 2026-03-20T18:06:24.652 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.652+0000 7ff72252a900 20 add_watcher() i=6 2026-03-20T18:06:24.652 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.653+0000 7ff72252a900 20 add_watcher() i=2 2026-03-20T18:06:24.652 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.653+0000 7ff72252a900 2 all 8 watchers are set, enabling cache 2026-03-20T18:06:24.654 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.655+0000 7ff72252a900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:06:24.655 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.655+0000 7ff72252a900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:06:24.655 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.655+0000 7ff72252a900 5 note: GC not initialized 2026-03-20T18:06:24.655 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.655+0000 7ff6d0fe1640 20 reqs_thread_entry: start 2026-03-20T18:06:24.701 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.701+0000 7ff72252a900 20 init_complete bucket index max shards: 11 2026-03-20T18:06:24.701 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.701+0000 7ff72252a900 20 Filter name: none 2026-03-20T18:06:24.701 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.701+0000 7ff6827fc640 20 reqs_thread_entry: start 2026-03-20T18:06:24.701 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.702+0000 7ff72252a900 20 rados->read ofs=0 len=0 2026-03-20T18:06:24.702 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.702+0000 7ff72252a900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:06:24.707 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.707+0000 7ff72252a900 20 remove_watcher() i=0 2026-03-20T18:06:24.707 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.707+0000 7ff72252a900 2 removed watcher, disabling cache 2026-03-20T18:06:24.707 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.707+0000 7ff72252a900 20 remove_watcher() i=5 2026-03-20T18:06:24.707 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.707+0000 7ff72252a900 20 remove_watcher() i=3 2026-03-20T18:06:24.707 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.708+0000 7ff72252a900 20 remove_watcher() i=7 2026-03-20T18:06:24.707 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.708+0000 7ff72252a900 20 remove_watcher() i=4 2026-03-20T18:06:24.707 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.708+0000 7ff72252a900 20 remove_watcher() i=6 2026-03-20T18:06:24.707 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.708+0000 7ff72252a900 20 remove_watcher() i=1 2026-03-20T18:06:24.708 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:24.708+0000 7ff72252a900 20 remove_watcher() i=2 2026-03-20T18:06:29.758 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:06:29.758 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:06:29.764 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.764+0000 7ff051d2f900 -1 asok(0x55f6de8945c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.69251.asok': (13) Permission denied 2026-03-20T18:06:29.783 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.783+0000 7ff051d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:06:29.784 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.784+0000 7ff051d2f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:29.784 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.784+0000 7ff051d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:06:29.784 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.785+0000 7ff051d2f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:29.784 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.785+0000 7ff051d2f900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:06:29.784 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.785+0000 7ff051d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:06:29.785 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.785+0000 7ff051d2f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:29.785 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.785+0000 7ff051d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:06:29.785 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.786+0000 7ff051d2f900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:06:29.786 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.786+0000 7ff051d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:06:29.787 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.787+0000 7ff051d2f900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:06:29.787 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.787+0000 7ff051d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:06:29.787 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.787+0000 7ff051d2f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:29.787 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.787+0000 7ff051d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:06:29.787 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.788+0000 7ff051d2f900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:06:29.787 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.788+0000 7ff051d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:06:29.788 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.788+0000 7ff051d2f900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:06:29.788 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.788+0000 7ff051d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.789+0000 7ff051d2f900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.789+0000 7ff051d2f900 20 zone MyZone found 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.789+0000 7ff051d2f900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.789+0000 7ff051d2f900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.789+0000 7ff051d2f900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.789+0000 7ff051d2f900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.789+0000 7ff051d2f900 20 period zonegroup name MyZoneGroup 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.789+0000 7ff051d2f900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.789+0000 7ff051d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.790+0000 7ff051d2f900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.790+0000 7ff051d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.790+0000 7ff051d2f900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.790+0000 7ff051d2f900 20 started sync module instance, tier type = 2026-03-20T18:06:29.789 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.790+0000 7ff051d2f900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:06:29.796 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.795+0000 7ff051d2f900 20 add_watcher() i=3 2026-03-20T18:06:29.796 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.796+0000 7ff051d2f900 20 add_watcher() i=1 2026-03-20T18:06:29.796 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.797+0000 7ff051d2f900 20 add_watcher() i=5 2026-03-20T18:06:29.796 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.797+0000 7ff051d2f900 20 add_watcher() i=6 2026-03-20T18:06:29.796 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.797+0000 7ff051d2f900 20 add_watcher() i=2 2026-03-20T18:06:29.796 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.797+0000 7ff051d2f900 20 add_watcher() i=0 2026-03-20T18:06:29.796 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.797+0000 7ff051d2f900 20 add_watcher() i=4 2026-03-20T18:06:29.796 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.797+0000 7ff051d2f900 20 add_watcher() i=7 2026-03-20T18:06:29.796 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.797+0000 7ff051d2f900 2 all 8 watchers are set, enabling cache 2026-03-20T18:06:29.799 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.799+0000 7ff051d2f900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:06:29.799 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.799+0000 7ff051d2f900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:06:29.799 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.799+0000 7ff051d2f900 5 note: GC not initialized 2026-03-20T18:06:29.799 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.799+0000 7fefb3fff640 20 reqs_thread_entry: start 2026-03-20T18:06:29.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.846+0000 7ff051d2f900 20 init_complete bucket index max shards: 11 2026-03-20T18:06:29.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.846+0000 7ff051d2f900 20 Filter name: none 2026-03-20T18:06:29.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.846+0000 7fefb1ffb640 20 reqs_thread_entry: start 2026-03-20T18:06:29.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.847+0000 7ff051d2f900 20 rados->read ofs=0 len=0 2026-03-20T18:06:29.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.847+0000 7ff051d2f900 20 rados_obj.operate() r=0 bl.length=262 2026-03-20T18:06:29.851 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.852+0000 7ff051d2f900 20 remove_watcher() i=4 2026-03-20T18:06:29.851 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.852+0000 7ff051d2f900 2 removed watcher, disabling cache 2026-03-20T18:06:29.851 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.852+0000 7ff051d2f900 20 remove_watcher() i=3 2026-03-20T18:06:29.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.853+0000 7ff051d2f900 20 remove_watcher() i=0 2026-03-20T18:06:29.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.853+0000 7ff051d2f900 20 remove_watcher() i=7 2026-03-20T18:06:29.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.853+0000 7ff051d2f900 20 remove_watcher() i=6 2026-03-20T18:06:29.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.853+0000 7ff051d2f900 20 remove_watcher() i=5 2026-03-20T18:06:29.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.853+0000 7ff051d2f900 20 remove_watcher() i=1 2026-03-20T18:06:29.852 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:06:29.853+0000 7ff051d2f900 20 remove_watcher() i=2 2026-03-20T18:06:40.003 INFO:teuthology.orchestra.run.vm01.stderr:ok 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stderr:test pushing persistent notification of large notifications ... ok 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stderr: 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stderr:---------------------------------------- 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stderr:Ran 13 tests in 274.216s 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stderr: 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stderr:OK 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:wait for 5sec for the messages... 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:wait for 5sec for the messages... 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:wait for the messages... 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:wait for the messages... 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:wait for 5sec for the messages... 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:wait for 5sec for the messages... 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:wait for 5sec for the messages... 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:wait for 5sec for the messages... 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:wait for 5sec for the messages... 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:version ok: QFHGvCaNasx.30c1RDx02Uu7C97Hkk0 in: ['QFHGvCaNasx.30c1RDx02Uu7C97Hkk0', 'XkpRKz60pSDB4n-HpP6jS69EVTWr0PQ', None] 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:version ok: XkpRKz60pSDB4n-HpP6jS69EVTWr0PQ in: ['QFHGvCaNasx.30c1RDx02Uu7C97Hkk0', 'XkpRKz60pSDB4n-HpP6jS69EVTWr0PQ', None] 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:version mismatch: Atd.NOGW04n5mHCQPEMY8QCLoLzjuy- not in: ['QFHGvCaNasx.30c1RDx02Uu7C97Hkk0', 'XkpRKz60pSDB4n-HpP6jS69EVTWr0PQ', None] 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:wait for 5sec for the messages... 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:version ok: SYWaCWRgP4shk-G8P3JgUf-mjB35fVq in: ['ScOsSCkP5mf0.fIdOY1pc91njSBI2IG', 'Jd-0lehLheUNkVIN.Xa.sDU1wxOB5mi', 'SYWaCWRgP4shk-G8P3JgUf-mjB35fVq'] 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:version ok: SYWaCWRgP4shk-G8P3JgUf-mjB35fVq in: ['ScOsSCkP5mf0.fIdOY1pc91njSBI2IG', 'Jd-0lehLheUNkVIN.Xa.sDU1wxOB5mi', 'SYWaCWRgP4shk-G8P3JgUf-mjB35fVq'] 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:version ok: Jd-0lehLheUNkVIN.Xa.sDU1wxOB5mi in: ['ScOsSCkP5mf0.fIdOY1pc91njSBI2IG', 'Jd-0lehLheUNkVIN.Xa.sDU1wxOB5mi', 'SYWaCWRgP4shk-G8P3JgUf-mjB35fVq'] 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:version ok: Jd-0lehLheUNkVIN.Xa.sDU1wxOB5mi in: ['ScOsSCkP5mf0.fIdOY1pc91njSBI2IG', 'Jd-0lehLheUNkVIN.Xa.sDU1wxOB5mi', 'SYWaCWRgP4shk-G8P3JgUf-mjB35fVq'] 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:version ok: ScOsSCkP5mf0.fIdOY1pc91njSBI2IG in: ['ScOsSCkP5mf0.fIdOY1pc91njSBI2IG', 'Jd-0lehLheUNkVIN.Xa.sDU1wxOB5mi', 'SYWaCWRgP4shk-G8P3JgUf-mjB35fVq'] 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:version ok: ScOsSCkP5mf0.fIdOY1pc91njSBI2IG in: ['ScOsSCkP5mf0.fIdOY1pc91njSBI2IG', 'Jd-0lehLheUNkVIN.Xa.sDU1wxOB5mi', 'SYWaCWRgP4shk-G8P3JgUf-mjB35fVq'] 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:average time for creation + async http notification is: 83.81478786468506 milliseconds 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:average time for deletion + async http notification is: 12.51535415649414 milliseconds 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:average time for creation + async http notification is: 88.47059965133667 milliseconds 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:average time for deletion + async http notification is: 12.28945255279541 milliseconds 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:average time for creation + async http notification is: 80.68635940551758 milliseconds 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:wait for 40sec for the messages... 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:average time for deletion + async http notification is: 12.201919555664062 milliseconds 2026-03-20T18:08:09.670 INFO:teuthology.orchestra.run.vm01.stdout:wait for 40sec for the messages... 2026-03-20T18:08:09.767 INFO:tasks.notification_tests:Removing bn-tests.conf file... 2026-03-20T18:08:09.767 DEBUG:teuthology.orchestra.run.vm01:> rm -f /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf 2026-03-20T18:08:09.780 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2026-03-20T18:08:09.853 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:08:09.853 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:08:09.856 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.857+0000 7f6805f24900 -1 asok(0x55556a466540) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.0.69582.asok': (13) Permission denied 2026-03-20T18:08:09.870 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.871+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.871 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.871+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:08:09.871 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.871+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.871 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.872+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:08:09.871 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.872+0000 7f6805f24900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:08:09.871 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.872+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.872 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.872+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:08:09.872 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.872+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.872 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.873+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:08:09.872 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.873+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.873 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.874+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:08:09.873 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.874+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.873 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.874+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:08:09.873 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.874+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.874 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.874+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:08:09.874 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.874+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.874 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.875+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:08:09.874 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.875+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.875 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:08:09.875 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 20 zone MyZone found 2026-03-20T18:08:09.875 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:08:09.875 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:08:09.875 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:08:09.875 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:08:09.875 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 20 period zonegroup name MyZoneGroup 2026-03-20T18:08:09.875 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:08:09.875 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.876 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:08:09.876 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.876 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:08:09.876 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 20 started sync module instance, tier type = 2026-03-20T18:08:09.876 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.876+0000 7f6805f24900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:08:09.879 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.880+0000 7f6805f24900 20 add_watcher() i=3 2026-03-20T18:08:09.879 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.880+0000 7f6805f24900 20 add_watcher() i=2 2026-03-20T18:08:09.882 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.883+0000 7f6805f24900 20 add_watcher() i=5 2026-03-20T18:08:09.882 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.883+0000 7f6805f24900 20 add_watcher() i=7 2026-03-20T18:08:09.882 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.883+0000 7f6805f24900 20 add_watcher() i=6 2026-03-20T18:08:09.883 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.884+0000 7f6805f24900 20 add_watcher() i=4 2026-03-20T18:08:09.883 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.884+0000 7f6805f24900 20 add_watcher() i=0 2026-03-20T18:08:09.883 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.884+0000 7f6805f24900 20 add_watcher() i=1 2026-03-20T18:08:09.883 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.884+0000 7f6805f24900 2 all 8 watchers are set, enabling cache 2026-03-20T18:08:09.885 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.886+0000 7f6805f24900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:08:09.885 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.886+0000 7f6805f24900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:08:09.915 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.916+0000 7f67b4fe1640 20 reqs_thread_entry: start 2026-03-20T18:08:09.952 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.952+0000 7f6805f24900 20 init_complete bucket index max shards: 11 2026-03-20T18:08:09.952 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.952+0000 7f6805f24900 20 Filter name: none 2026-03-20T18:08:09.952 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.952+0000 7f67adffb640 20 reqs_thread_entry: start 2026-03-20T18:08:09.952 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 10 cache get: name=MyZone.rgw.meta+users.uid+foo.client.0 : miss 2026-03-20T18:08:09.952 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.952 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=443 2026-03-20T18:08:09.952 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 10 cache put: name=MyZone.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2026-03-20T18:08:09.952 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 10 adding MyZone.rgw.meta+users.uid+foo.client.0 to cache LRU end 2026-03-20T18:08:09.952 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-03-20T18:08:09.953 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 10 cache get: name=MyZone.rgw.meta+root+.bucket.meta.ephezu-4:c5d78a84-c400-4c18-a0be-5d12df626549.4659.4 : miss 2026-03-20T18:08:09.953 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.953 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=416 2026-03-20T18:08:09.953 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 10 cache put: name=MyZone.rgw.meta+root+.bucket.meta.ephezu-4:c5d78a84-c400-4c18-a0be-5d12df626549.4659.4 info.flags=0x17 2026-03-20T18:08:09.953 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 10 adding MyZone.rgw.meta+root+.bucket.meta.ephezu-4:c5d78a84-c400-4c18-a0be-5d12df626549.4659.4 to cache LRU end 2026-03-20T18:08:09.953 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-03-20T18:08:09.953 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 10 updating xattr: name=user.rgw.acl bl.length()=185 2026-03-20T18:08:09.953 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.953+0000 7f6805f24900 10 chain_cache_entry: cache_locator=MyZone.rgw.meta+root+.bucket.meta.ephezu-4:c5d78a84-c400-4c18-a0be-5d12df626549.4659.4 2026-03-20T18:08:09.956 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.956+0000 7f6805f24900 20 list_objects_ordered: starting attempt 1 2026-03-20T18:08:09.956 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.956+0000 7f6805f24900 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2026-03-20T18:08:09.957 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.957+0000 7f6805f24900 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2026-03-20T18:08:09.957 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.957+0000 7f6805f24900 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2026-03-20T18:08:09.957 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.957+0000 7f6805f24900 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2026-03-20T18:08:09.957 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.957+0000 7f6805f24900 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2026-03-20T18:08:09.957 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.957+0000 7f6805f24900 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":ephezu-4[c5d78a84-c400-4c18-a0be-5d12df626549.4659.4])"; uploads.size()=0; is_truncated=0 2026-03-20T18:08:09.958 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.959+0000 7f6805f24900 10 cache get: name=MyZone.rgw.meta+root+ephezu-4 : miss 2026-03-20T18:08:09.958 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.959+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.959 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.959+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=186 2026-03-20T18:08:09.959 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.959+0000 7f6805f24900 10 cache put: name=MyZone.rgw.meta+root+ephezu-4 info.flags=0x11 2026-03-20T18:08:09.959 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.959+0000 7f6805f24900 10 adding MyZone.rgw.meta+root+ephezu-4 to cache LRU end 2026-03-20T18:08:09.960 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.960+0000 7f6805f24900 10 removing MyZone.rgw.meta+root+ephezu-4 from cache 2026-03-20T18:08:09.960 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.960+0000 7f6805f24900 10 distributing notification oid=MyZone.rgw.control:notify.3 cni=[op: 1, obj: MyZone.rgw.meta:root:ephezu-4, ofs0, ns] 2026-03-20T18:08:09.960 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.960+0000 7f67dd7fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 197568495635 cookie 93825349906480 notifier 4988 bl.length()=151 2026-03-20T18:08:09.963 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.963+0000 7f6805f24900 10 removing MyZone.rgw.meta+root+.bucket.meta.ephezu-4:c5d78a84-c400-4c18-a0be-5d12df626549.4659.4 from cache 2026-03-20T18:08:09.963 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.963+0000 7f6805f24900 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 1, obj: MyZone.rgw.meta:root:.bucket.meta.ephezu-4:c5d78a84-c400-4c18-a0be-5d12df626549.4659.4, ofs0, ns] 2026-03-20T18:08:09.963 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.964+0000 7f67dd7fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 197568495659 cookie 93825348984176 notifier 4988 bl.length()=208 2026-03-20T18:08:09.967 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.968+0000 7f6805f24900 10 cache get: name=MyZone.rgw.log++pubsub..bucket.ephezu-4/c5d78a84-c400-4c18-a0be-5d12df626549.4659.4 : miss 2026-03-20T18:08:09.967 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.968+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.967 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.968+0000 7f6805f24900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:08:09.967 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.968+0000 7f6805f24900 10 cache put: name=MyZone.rgw.log++pubsub..bucket.ephezu-4/c5d78a84-c400-4c18-a0be-5d12df626549.4659.4 info.flags=0x0 2026-03-20T18:08:09.967 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.968+0000 7f6805f24900 10 adding MyZone.rgw.log++pubsub..bucket.ephezu-4/c5d78a84-c400-4c18-a0be-5d12df626549.4659.4 to cache LRU end 2026-03-20T18:08:09.969 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.969+0000 7f6805f24900 10 cache get: name=MyZone.rgw.meta+root+.bucket.meta.ephezu-8:c5d78a84-c400-4c18-a0be-5d12df626549.4659.8 : miss 2026-03-20T18:08:09.969 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.969+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.969 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.970+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=416 2026-03-20T18:08:09.969 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.970+0000 7f6805f24900 10 cache put: name=MyZone.rgw.meta+root+.bucket.meta.ephezu-8:c5d78a84-c400-4c18-a0be-5d12df626549.4659.8 info.flags=0x17 2026-03-20T18:08:09.969 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.970+0000 7f6805f24900 10 adding MyZone.rgw.meta+root+.bucket.meta.ephezu-8:c5d78a84-c400-4c18-a0be-5d12df626549.4659.8 to cache LRU end 2026-03-20T18:08:09.969 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.970+0000 7f6805f24900 10 updating xattr: name=ceph.objclass.version bl.length()=42 2026-03-20T18:08:09.969 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.970+0000 7f6805f24900 10 updating xattr: name=user.rgw.acl bl.length()=185 2026-03-20T18:08:09.969 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.970+0000 7f6805f24900 10 chain_cache_entry: cache_locator=MyZone.rgw.meta+root+.bucket.meta.ephezu-8:c5d78a84-c400-4c18-a0be-5d12df626549.4659.8 2026-03-20T18:08:09.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.972+0000 7f6805f24900 10 cls_bucket_list_unordered: got copy_of_foo[o98fPRWZfnBSJJKT.95WecCwCSC8bAi] 2026-03-20T18:08:09.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.972+0000 7f6805f24900 10 cls_bucket_list_unordered: got copy_of_foo[Atd.NOGW04n5mHCQPEMY8QCLoLzjuy-] 2026-03-20T18:08:09.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.972+0000 7f6805f24900 20 bucket index object: :.dir.c5d78a84-c400-4c18-a0be-5d12df626549.4659.8.9 2026-03-20T18:08:09.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.973+0000 7f6805f24900 20 get_obj_state: octx=0x55556aa44440 obj=ephezu-8:copy_of_foo state=0x55556aa31f68 s->prefetch_data=0 2026-03-20T18:08:09.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.973+0000 7f6805f24900 20 get_obj_state: setting s->obj_tag to 08ht9lozsrfvh30kweiw0mir59tq141z 2026-03-20T18:08:09.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.973+0000 7f6805f24900 20 get_obj_state_impl: setting s->olh_tag to 99j5qb5dru4du757t0y1bxefhurq1qvc 2026-03-20T18:08:09.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.973+0000 7f6805f24900 20 bucket_index_guard_olh_op(): olh_state.olh_tag=99j5qb5dru4du757t0y1bxefhurq1qvc 2026-03-20T18:08:09.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.974+0000 7f6805f24900 20 bucket index object: :.dir.c5d78a84-c400-4c18-a0be-5d12df626549.4659.8.9 2026-03-20T18:08:09.975 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.975+0000 7f6805f24900 20 bucket index object: :.dir.c5d78a84-c400-4c18-a0be-5d12df626549.4659.8.9 2026-03-20T18:08:09.975 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.976+0000 7f6805f24900 20 olh_log_entry: epoch=4 op=1 key=copy_of_foo[Atd.NOGW04n5mHCQPEMY8QCLoLzjuy-] 2026-03-20T18:08:09.975 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.976+0000 7f6805f24900 20 apply_olh_log applying key=copy_of_foo[Atd.NOGW04n5mHCQPEMY8QCLoLzjuy-] epoch=4 delete_marker=0 over current=copy_of_foo[o98fPRWZfnBSJJKT.95WecCwCSC8bAi] epoch=3 delete_marker=1 2026-03-20T18:08:09.976 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.978+0000 7f6805f24900 20 bucket index object: :.dir.c5d78a84-c400-4c18-a0be-5d12df626549.4659.8.9 2026-03-20T18:08:09.976 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.978+0000 7f6805f24900 20 bucket index object: :.dir.c5d78a84-c400-4c18-a0be-5d12df626549.4659.8.9 2026-03-20T18:08:09.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.978+0000 7f6805f24900 20 bucket index object: :.dir.c5d78a84-c400-4c18-a0be-5d12df626549.4659.8.9 2026-03-20T18:08:09.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.978+0000 7f6805f24900 20 bucket index object: :.dir.c5d78a84-c400-4c18-a0be-5d12df626549.4659.8.9 2026-03-20T18:08:09.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.978+0000 7f6805f24900 20 get_obj_state: octx=0x55556aa44440 obj=ephezu-8:copy_of_foo state=0x55556aa31f68 s->prefetch_data=0 2026-03-20T18:08:09.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.979+0000 7f6805f24900 20 get_obj_state: setting s->obj_tag to 08ht9lozsrfvh30kweiw0mir59tq141z 2026-03-20T18:08:09.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.979+0000 7f6805f24900 20 get_obj_state_impl: setting s->olh_tag to 99j5qb5dru4du757t0y1bxefhurq1qvc 2026-03-20T18:08:09.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.979+0000 7f6805f24900 20 bucket_index_guard_olh_op(): olh_state.olh_tag=99j5qb5dru4du757t0y1bxefhurq1qvc 2026-03-20T18:08:09.979 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.980+0000 7f6805f24900 20 bucket index object: :.dir.c5d78a84-c400-4c18-a0be-5d12df626549.4659.8.9 2026-03-20T18:08:09.980 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.981+0000 7f6805f24900 20 bucket index object: :.dir.c5d78a84-c400-4c18-a0be-5d12df626549.4659.8.9 2026-03-20T18:08:09.981 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.981+0000 7f6805f24900 20 olh_log_entry: epoch=5 op=2 key=copy_of_foo[] 2026-03-20T18:08:09.981 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.981+0000 7f6805f24900 20 olh_log_entry: epoch=5 op=3 key=copy_of_foo[Atd.NOGW04n5mHCQPEMY8QCLoLzjuy-] 2026-03-20T18:08:09.981 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.981+0000 7f6805f24900 20 get_obj_state: octx=0x55556aa44440 obj=ephezu-8:_:Atd.NOGW04n5mHCQPEMY8QCLoLzjuy-_copy_of_foo state=0x55556aa426c8 s->prefetch_data=0 2026-03-20T18:08:09.981 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.982+0000 7f6805f24900 10 manifest: total_size = 5 2026-03-20T18:08:09.981 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.982+0000 7f6805f24900 20 get_obj_state: setting s->obj_tag to _rewBbZ_NXkixdG9G2PAk-Mc6QF1uAYz 2026-03-20T18:08:09.981 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.982+0000 7f6805f24900 20 bucket index object: :.dir.c5d78a84-c400-4c18-a0be-5d12df626549.4659.8.9 2026-03-20T18:08:09.983 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.984+0000 7f6805f24900 20 get_obj_state: octx=0x55556aa44440 obj=ephezu-8:_:Atd.NOGW04n5mHCQPEMY8QCLoLzjuy-_copy_of_foo state=0x55556aa426c8 s->prefetch_data=0 2026-03-20T18:08:09.984 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.985+0000 7f6805f24900 20 get_obj_state: octx=0x55556aa44440 obj=ephezu-8:copy_of_foo state=0x55556aa31f68 s->prefetch_data=0 2026-03-20T18:08:09.985 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.985+0000 7f67dd7fa640 20 handle_completion(): completion ok for obj=copy_of_foo[Atd.NOGW04n5mHCQPEMY8QCLoLzjuy-] 2026-03-20T18:08:09.985 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.986+0000 7f6805f24900 20 bucket index object: :.dir.c5d78a84-c400-4c18-a0be-5d12df626549.4659.8.9 2026-03-20T18:08:09.986 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.987+0000 7f6805f24900 20 bucket index object: :.dir.c5d78a84-c400-4c18-a0be-5d12df626549.4659.8.9 2026-03-20T18:08:09.987 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.987+0000 7f6805f24900 20 list_objects_ordered: starting attempt 1 2026-03-20T18:08:09.987 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.987+0000 7f6805f24900 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2026-03-20T18:08:09.987 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.988+0000 7f6805f24900 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2026-03-20T18:08:09.988 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.988+0000 7f6805f24900 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2026-03-20T18:08:09.988 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.988+0000 7f6805f24900 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2026-03-20T18:08:09.988 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.988+0000 7f6805f24900 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2026-03-20T18:08:09.988 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.988+0000 7f6805f24900 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":ephezu-8[c5d78a84-c400-4c18-a0be-5d12df626549.4659.8])"; uploads.size()=0; is_truncated=0 2026-03-20T18:08:09.989 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.990+0000 7f6805f24900 10 cache get: name=MyZone.rgw.meta+root+ephezu-8 : miss 2026-03-20T18:08:09.989 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.990+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.989 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.990+0000 7f6805f24900 20 rados_obj.operate() r=0 bl.length=186 2026-03-20T18:08:09.989 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.990+0000 7f6805f24900 10 cache put: name=MyZone.rgw.meta+root+ephezu-8 info.flags=0x11 2026-03-20T18:08:09.989 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.990+0000 7f6805f24900 10 adding MyZone.rgw.meta+root+ephezu-8 to cache LRU end 2026-03-20T18:08:09.990 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.992+0000 7f6805f24900 10 removing MyZone.rgw.meta+root+ephezu-8 from cache 2026-03-20T18:08:09.990 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.992+0000 7f6805f24900 10 distributing notification oid=MyZone.rgw.control:notify.3 cni=[op: 1, obj: MyZone.rgw.meta:root:ephezu-8, ofs0, ns] 2026-03-20T18:08:09.991 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.992+0000 7f67dd7fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 197568495636 cookie 93825349906480 notifier 4988 bl.length()=151 2026-03-20T18:08:09.993 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.994+0000 7f6805f24900 10 removing MyZone.rgw.meta+root+.bucket.meta.ephezu-8:c5d78a84-c400-4c18-a0be-5d12df626549.4659.8 from cache 2026-03-20T18:08:09.993 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.994+0000 7f6805f24900 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 1, obj: MyZone.rgw.meta:root:.bucket.meta.ephezu-8:c5d78a84-c400-4c18-a0be-5d12df626549.4659.8, ofs0, ns] 2026-03-20T18:08:09.994 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.994+0000 7f67dd7fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 197568495660 cookie 93825348984176 notifier 4988 bl.length()=208 2026-03-20T18:08:09.998 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.998+0000 7f6805f24900 10 cache get: name=MyZone.rgw.log++pubsub..bucket.ephezu-8/c5d78a84-c400-4c18-a0be-5d12df626549.4659.8 : miss 2026-03-20T18:08:09.998 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.998+0000 7f6805f24900 20 rados->read ofs=0 len=0 2026-03-20T18:08:09.998 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.999+0000 7f6805f24900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:08:09.998 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.999+0000 7f6805f24900 10 cache put: name=MyZone.rgw.log++pubsub..bucket.ephezu-8/c5d78a84-c400-4c18-a0be-5d12df626549.4659.8 info.flags=0x0 2026-03-20T18:08:09.998 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:09.999+0000 7f6805f24900 10 adding MyZone.rgw.log++pubsub..bucket.ephezu-8/c5d78a84-c400-4c18-a0be-5d12df626549.4659.8 to cache LRU end 2026-03-20T18:08:10.000 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.001+0000 7f6805f24900 10 removing key index: QHRUKFLLXTHPEBHYWCSR 2026-03-20T18:08:10.001 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.001+0000 7f6805f24900 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 1, obj: MyZone.rgw.meta:users.keys:QHRUKFLLXTHPEBHYWCSR, ofs0, ns] 2026-03-20T18:08:10.001 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.002+0000 7f67dd7fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 197568495661 cookie 93825348984176 notifier 4988 bl.length()=169 2026-03-20T18:08:10.003 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.003+0000 7f6805f24900 10 removing email index: 2026-03-20T18:08:10.003 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.003+0000 7f6805f24900 10 removing user buckets index 2026-03-20T18:08:10.003 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.005+0000 7f6805f24900 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 1, obj: MyZone.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2026-03-20T18:08:10.004 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.005+0000 7f67dd7fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 197568495662 cookie 93825348984176 notifier 4988 bl.length()=168 2026-03-20T18:08:10.006 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.007+0000 7f6805f24900 10 removing user index: foo.client.0 2026-03-20T18:08:10.008 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.009+0000 7f6805f24900 10 removing MyZone.rgw.meta+users.uid+foo.client.0 from cache 2026-03-20T18:08:10.008 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.009+0000 7f6805f24900 10 distributing notification oid=MyZone.rgw.control:notify.0 cni=[op: 1, obj: MyZone.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2026-03-20T18:08:10.009 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.009+0000 7f67ddffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 197568495619 cookie 93825350056528 notifier 4988 bl.length()=160 2026-03-20T18:08:10.015 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.015+0000 7f6805f24900 20 remove_watcher() i=3 2026-03-20T18:08:10.015 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.015+0000 7f6805f24900 2 removed watcher, disabling cache 2026-03-20T18:08:10.015 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.015+0000 7f6805f24900 20 remove_watcher() i=2 2026-03-20T18:08:10.015 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.015+0000 7f6805f24900 20 remove_watcher() i=0 2026-03-20T18:08:10.015 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.015+0000 7f6805f24900 20 remove_watcher() i=4 2026-03-20T18:08:10.015 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.016+0000 7f6805f24900 20 remove_watcher() i=5 2026-03-20T18:08:10.015 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.016+0000 7f6805f24900 20 remove_watcher() i=6 2026-03-20T18:08:10.015 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.016+0000 7f6805f24900 20 remove_watcher() i=1 2026-03-20T18:08:10.015 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:10.016+0000 7f6805f24900 20 remove_watcher() i=7 2026-03-20T18:08:10.021 INFO:tasks.notification_tests:Pre-processing completed... 2026-03-20T18:08:10.021 DEBUG:teuthology.orchestra.run.vm01:> rm -rf /home/ubuntu/.aws/models/s3/2006-03-01/service-2.sdk-extras.json 2026-03-20T18:08:10.076 DEBUG:teuthology.orchestra.run.vm01:> cd /home/ubuntu/ && rmdir -p .aws/models/s3/2006-03-01/ 2026-03-20T18:08:10.131 INFO:tasks.notification_tests:Removing bucket-notifications-tests... 2026-03-20T18:08:10.131 DEBUG:teuthology.orchestra.run.vm01:> rm -rf /home/ubuntu/cephtest/ceph 2026-03-20T18:08:10.628 DEBUG:teuthology.run_tasks:Unwinding manager notification-tests 2026-03-20T18:08:10.631 DEBUG:teuthology.run_tasks:Unwinding manager rabbitmq 2026-03-20T18:08:10.632 INFO:tasks.rabbitmq:Stopping RabbitMQ Service... 2026-03-20T18:08:10.632 DEBUG:teuthology.orchestra.run.vm01:> sudo systemctl stop rabbitmq-server 2026-03-20T18:08:13.042 INFO:tasks.rabbitmq:Removing packaged dependencies of RabbitMQ... 2026-03-20T18:08:13.042 DEBUG:teuthology.orchestra.run.vm01:> sudo dnf -y remove rabbitmq-server.noarch 2026-03-20T18:08:13.211 INFO:teuthology.orchestra.run.vm01.stdout:Dependencies resolved. 2026-03-20T18:08:13.211 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:08:13.211 INFO:teuthology.orchestra.run.vm01.stdout: Package Arch Version Repository Size 2026-03-20T18:08:13.212 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:08:13.212 INFO:teuthology.orchestra.run.vm01.stdout:Removing: 2026-03-20T18:08:13.212 INFO:teuthology.orchestra.run.vm01.stdout: rabbitmq-server noarch 3.13.7-1.el8 @rabbitmq_rabbitmq-server 24 M 2026-03-20T18:08:13.212 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:08:13.212 INFO:teuthology.orchestra.run.vm01.stdout:Transaction Summary 2026-03-20T18:08:13.212 INFO:teuthology.orchestra.run.vm01.stdout:================================================================================ 2026-03-20T18:08:13.212 INFO:teuthology.orchestra.run.vm01.stdout:Remove 1 Package 2026-03-20T18:08:13.212 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:08:13.212 INFO:teuthology.orchestra.run.vm01.stdout:Freed space: 24 M 2026-03-20T18:08:13.212 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction check 2026-03-20T18:08:13.214 INFO:teuthology.orchestra.run.vm01.stdout:Transaction check succeeded. 2026-03-20T18:08:13.214 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction test 2026-03-20T18:08:13.234 INFO:teuthology.orchestra.run.vm01.stdout:Transaction test succeeded. 2026-03-20T18:08:13.234 INFO:teuthology.orchestra.run.vm01.stdout:Running transaction 2026-03-20T18:08:13.272 INFO:teuthology.orchestra.run.vm01.stdout: Preparing : 1/1 2026-03-20T18:08:13.330 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: rabbitmq-server-3.13.7-1.el8.noarch 1/1 2026-03-20T18:08:13.330 INFO:teuthology.orchestra.run.vm01.stdout: Erasing : rabbitmq-server-3.13.7-1.el8.noarch 1/1 2026-03-20T18:08:13.428 INFO:teuthology.orchestra.run.vm01.stdout: Running scriptlet: rabbitmq-server-3.13.7-1.el8.noarch 1/1 2026-03-20T18:08:13.471 INFO:teuthology.orchestra.run.vm01.stdout: Verifying : rabbitmq-server-3.13.7-1.el8.noarch 1/1 2026-03-20T18:08:13.471 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:08:13.471 INFO:teuthology.orchestra.run.vm01.stdout:Removed: 2026-03-20T18:08:13.471 INFO:teuthology.orchestra.run.vm01.stdout: rabbitmq-server-3.13.7-1.el8.noarch 2026-03-20T18:08:13.471 INFO:teuthology.orchestra.run.vm01.stdout: 2026-03-20T18:08:13.471 INFO:teuthology.orchestra.run.vm01.stdout:Complete! 2026-03-20T18:08:13.497 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2026-03-20T18:08:13.499 DEBUG:tasks.rgw.client.0:waiting for process to exit 2026-03-20T18:08:13.499 INFO:teuthology.orchestra.run:waiting for 300 2026-03-20T18:08:13.503 INFO:tasks.rgw.client.0.vm01.stdout:2026-03-20T18:08:13.502+0000 1183c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/ceph.client.0.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground (PID: 63451) UID: 0 2026-03-20T18:08:13.507 INFO:tasks.rgw.client.0.vm01.stdout:2026-03-20T18:08:13.504+0000 bf16f40 -1 shutting down 2026-03-20T18:08:16.059 INFO:tasks.rgw.client.0.vm01.stderr:==00:00:06:48.517 63453== could not unlink /tmp/vgdb-pipe-from-vgdb-to-63453-by-root-on-??? 2026-03-20T18:08:16.059 INFO:tasks.rgw.client.0.vm01.stderr:==00:00:06:48.517 63453== could not unlink /tmp/vgdb-pipe-to-vgdb-from-63453-by-root-on-??? 2026-03-20T18:08:16.059 INFO:tasks.rgw.client.0.vm01.stderr:==00:00:06:48.517 63453== could not unlink /tmp/vgdb-pipe-shared-mem-vgdb-63453-by-root-on-??? 2026-03-20T18:08:16.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2026-03-20T18:08:19.601 INFO:tasks.rgw.client.0:Stopped 2026-03-20T18:08:19.601 DEBUG:teuthology.orchestra.run.vm01:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2026-03-20T18:08:19.617 DEBUG:teuthology.orchestra.run.vm01:> sudo rm -f /etc/ceph/vault-root-token 2026-03-20T18:08:19.681 DEBUG:teuthology.orchestra.run.vm01:> sudo rm -f /home/ubuntu/cephtest/url_file 2026-03-20T18:08:19.745 INFO:tasks.util.rgw:rgwadmin: client.0 : ['gc', 'process', '--include-all'] 2026-03-20T18:08:19.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'gc', 'process', '--include-all'] 2026-03-20T18:08:19.745 DEBUG:teuthology.orchestra.run.vm01:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc process --include-all 2026-03-20T18:08:19.818 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setuser ceph since I am not root 2026-03-20T18:08:19.818 INFO:teuthology.orchestra.run.vm01.stderr:ignoring --setgroup ceph since I am not root 2026-03-20T18:08:19.821 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.822+0000 7f40a2730900 -1 asok(0x55ae008f7600) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.0.70034.asok': (13) Permission denied 2026-03-20T18:08:19.836 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.836+0000 7f40a2730900 20 rados->read ofs=0 len=0 2026-03-20T18:08:19.836 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.837+0000 7f40a2730900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:08:19.836 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.837+0000 7f40a2730900 20 rados->read ofs=0 len=0 2026-03-20T18:08:19.837 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.838+0000 7f40a2730900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:08:19.837 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.838+0000 7f40a2730900 20 realm MyRealm 184e499c-426a-4683-9ccd-d819b580f6c1 2026-03-20T18:08:19.837 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.838+0000 7f40a2730900 20 rados->read ofs=0 len=0 2026-03-20T18:08:19.837 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.838+0000 7f40a2730900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:08:19.837 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.838+0000 7f40a2730900 20 rados->read ofs=0 len=0 2026-03-20T18:08:19.838 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.838+0000 7f40a2730900 20 rados_obj.operate() r=0 bl.length=10 2026-03-20T18:08:19.838 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.838+0000 7f40a2730900 20 rados->read ofs=0 len=0 2026-03-20T18:08:19.838 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.839+0000 7f40a2730900 20 rados_obj.operate() r=0 bl.length=1104 2026-03-20T18:08:19.838 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.839+0000 7f40a2730900 20 rados->read ofs=0 len=0 2026-03-20T18:08:19.839 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.839+0000 7f40a2730900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:08:19.839 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.839+0000 7f40a2730900 20 rados->read ofs=0 len=0 2026-03-20T18:08:19.839 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.839+0000 7f40a2730900 20 rados_obj.operate() r=0 bl.length=107 2026-03-20T18:08:19.839 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.840+0000 7f40a2730900 20 rados->read ofs=0 len=0 2026-03-20T18:08:19.839 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.840+0000 7f40a2730900 20 rados_obj.operate() r=0 bl.length=46 2026-03-20T18:08:19.839 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.840+0000 7f40a2730900 20 rados->read ofs=0 len=0 2026-03-20T18:08:19.840 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 20 rados_obj.operate() r=0 bl.length=1200 2026-03-20T18:08:19.840 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 20 zone MyZone found 2026-03-20T18:08:19.840 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 4 Realm: MyRealm (184e499c-426a-4683-9ccd-d819b580f6c1) 2026-03-20T18:08:19.840 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 4 ZoneGroup: MyZoneGroup (bef9bd2b-4b23-45c1-9e03-2dec3d64903d) 2026-03-20T18:08:19.840 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 4 Zone: MyZone (c5d78a84-c400-4c18-a0be-5d12df626549) 2026-03-20T18:08:19.840 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 4 using period configuration: 0caded40-e639-4de0-bbbe-6af6ee4fba80:1 2026-03-20T18:08:19.840 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 20 period zonegroup name MyZoneGroup 2026-03-20T18:08:19.840 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 20 using current period zonegroup MyZoneGroup 2026-03-20T18:08:19.840 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 20 rados->read ofs=0 len=0 2026-03-20T18:08:19.840 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:08:19.840 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 20 rados->read ofs=0 len=0 2026-03-20T18:08:19.841 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 20 rados_obj.operate() r=-2 bl.length=0 2026-03-20T18:08:19.841 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 20 started sync module instance, tier type = 2026-03-20T18:08:19.841 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.841+0000 7f40a2730900 20 started zone id=c5d78a84-c400-4c18-a0be-5d12df626549 (name=MyZone) with tier type = 2026-03-20T18:08:19.845 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.845+0000 7f40a2730900 20 add_watcher() i=2 2026-03-20T18:08:19.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.847+0000 7f40a2730900 20 add_watcher() i=7 2026-03-20T18:08:19.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.847+0000 7f40a2730900 20 add_watcher() i=5 2026-03-20T18:08:19.846 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.847+0000 7f40a2730900 20 add_watcher() i=1 2026-03-20T18:08:19.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.848+0000 7f40a2730900 20 add_watcher() i=6 2026-03-20T18:08:19.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.848+0000 7f40a2730900 20 add_watcher() i=3 2026-03-20T18:08:19.847 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.848+0000 7f40a2730900 20 add_watcher() i=4 2026-03-20T18:08:19.848 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.848+0000 7f40a2730900 20 add_watcher() i=0 2026-03-20T18:08:19.848 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.848+0000 7f40a2730900 2 all 8 watchers are set, enabling cache 2026-03-20T18:08:19.849 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.850+0000 7f40a2730900 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2026-03-20T18:08:19.849 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.850+0000 7f40a2730900 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2026-03-20T18:08:19.877 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.877+0000 7f4050fe1640 20 reqs_thread_entry: start 2026-03-20T18:08:19.915 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.915+0000 7f40a2730900 20 init_complete bucket index max shards: 11 2026-03-20T18:08:19.915 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.915+0000 7f40a2730900 20 Filter name: none 2026-03-20T18:08:19.915 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.916+0000 7f40027fc640 20 reqs_thread_entry: start 2026-03-20T18:08:19.915 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.916+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2026-03-20T18:08:19.916 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.917+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.916 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.917+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.917 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.917+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.917 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.918+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2026-03-20T18:08:19.920 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.920+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.920 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.921+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.920 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.921+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.921 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.921+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2026-03-20T18:08:19.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.923+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.923+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.922 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.923+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.924 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.924+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2026-03-20T18:08:19.925 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.926+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.926 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.926+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.926 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.926+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.927 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.927+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2026-03-20T18:08:19.928 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.929+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.928 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.929+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.928 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.929+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.929 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.930+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2026-03-20T18:08:19.930 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.931+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.930 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.931+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.931 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.931+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.932 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.932+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2026-03-20T18:08:19.933 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.934+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.933 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.934+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.933 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.934+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2026-03-20T18:08:19.933 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.934+0000 7f40a2730900 20 garbage collection: RGWGC::process iterating over entry tag='c5d78a84-c400-4c18-a0be-5d12df626549.4659.13055453534192554519', time=2026-03-20T20:03:53.455743+0000, chain.objs.size()=3 2026-03-20T18:08:19.933 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.934+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.2__multipart_multipart.2~isoEU12HHoYSMu-Ojijk9UCLTHykYDV.1 2026-03-20T18:08:19.933 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.934+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.2__shadow_multipart.2~isoEU12HHoYSMu-Ojijk9UCLTHykYDV.1_1 2026-03-20T18:08:19.933 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.934+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.2__shadow_multipart.2~isoEU12HHoYSMu-Ojijk9UCLTHykYDV.1_2 2026-03-20T18:08:19.935 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.936+0000 7f40a2730900 5 garbage collection: RGWGC::process removing entries, marker: 2026-03-20T18:08:19.937 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.938+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2026-03-20T18:08:19.938 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.939+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.939 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.939+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.939 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.940+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.940 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.940+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2026-03-20T18:08:19.941 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.941+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.941 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.941+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.941 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.942+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.942 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.943+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2026-03-20T18:08:19.943 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.944+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.944 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.944+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.944 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.945+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.945 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.945+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2026-03-20T18:08:19.946 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.947+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.946 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.947+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.946 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.947+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.947 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.948+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2026-03-20T18:08:19.948 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.949+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.948 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.949+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.949 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.949+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.949 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.950+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2026-03-20T18:08:19.951 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.951+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.951 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.952+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.951 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.952+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.952 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.953+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2026-03-20T18:08:19.953 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.954+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.953 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.954+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.954 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.954+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.955 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.955+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2026-03-20T18:08:19.956 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.957+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.956 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.957+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.956 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.957+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.957 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.958+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2026-03-20T18:08:19.958 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.959+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.959 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.959+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.959 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.959+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.960 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.960+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2026-03-20T18:08:19.960 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.961+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.961 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.961+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.961 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.961+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.963 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.963+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2026-03-20T18:08:19.964 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.964+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.964 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.965+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.964 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.965+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2026-03-20T18:08:19.964 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.965+0000 7f40a2730900 20 garbage collection: RGWGC::process iterating over entry tag='c5d78a84-c400-4c18-a0be-5d12df626549.4659.14828031263493846790', time=2026-03-20T20:04:38.053306+0000, chain.objs.size()=8 2026-03-20T18:08:19.964 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.965+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.6__multipart_multipart_foo.2~tzv3TiRJK5zrbwbOvuO8bgnNlXvY4F8.1 2026-03-20T18:08:19.964 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.965+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.6__shadow_multipart_foo.2~tzv3TiRJK5zrbwbOvuO8bgnNlXvY4F8.1_1 2026-03-20T18:08:19.964 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.965+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.6__multipart_multipart_foo.2~tzv3TiRJK5zrbwbOvuO8bgnNlXvY4F8.2 2026-03-20T18:08:19.964 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.965+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.6__shadow_multipart_foo.2~tzv3TiRJK5zrbwbOvuO8bgnNlXvY4F8.2_1 2026-03-20T18:08:19.965 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.965+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.6__multipart_multipart_foo.2~tzv3TiRJK5zrbwbOvuO8bgnNlXvY4F8.3 2026-03-20T18:08:19.965 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.965+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.6__shadow_multipart_foo.2~tzv3TiRJK5zrbwbOvuO8bgnNlXvY4F8.3_1 2026-03-20T18:08:19.965 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.965+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.6__multipart_multipart_foo.2~tzv3TiRJK5zrbwbOvuO8bgnNlXvY4F8.4 2026-03-20T18:08:19.965 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.965+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.6__shadow_multipart_foo.2~tzv3TiRJK5zrbwbOvuO8bgnNlXvY4F8.4_1 2026-03-20T18:08:19.967 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.968+0000 7f40a2730900 5 garbage collection: RGWGC::process removing entries, marker: 2026-03-20T18:08:19.969 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.970+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2026-03-20T18:08:19.970 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.971+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.970 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.971+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.971 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.971+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.971 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.973+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2026-03-20T18:08:19.972 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.973+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.973+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.974+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2026-03-20T18:08:19.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.974+0000 7f40a2730900 20 garbage collection: RGWGC::process iterating over entry tag='c5d78a84-c400-4c18-a0be-5d12df626549.4659.7012097376584253710', time=2026-03-20T20:04:21.076072+0000, chain.objs.size()=8 2026-03-20T18:08:19.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.974+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.5__multipart_multipart_foo.2~OxemP2sKXW07N2--FqAxVV5Yg72boh5.1 2026-03-20T18:08:19.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.974+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.5__shadow_multipart_foo.2~OxemP2sKXW07N2--FqAxVV5Yg72boh5.1_1 2026-03-20T18:08:19.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.974+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.5__multipart_multipart_foo.2~OxemP2sKXW07N2--FqAxVV5Yg72boh5.2 2026-03-20T18:08:19.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.974+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.5__shadow_multipart_foo.2~OxemP2sKXW07N2--FqAxVV5Yg72boh5.2_1 2026-03-20T18:08:19.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.974+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.5__multipart_multipart_foo.2~OxemP2sKXW07N2--FqAxVV5Yg72boh5.3 2026-03-20T18:08:19.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.974+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.5__shadow_multipart_foo.2~OxemP2sKXW07N2--FqAxVV5Yg72boh5.3_1 2026-03-20T18:08:19.973 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.974+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.5__multipart_multipart_foo.2~OxemP2sKXW07N2--FqAxVV5Yg72boh5.4 2026-03-20T18:08:19.974 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.974+0000 7f40a2730900 5 garbage collection: RGWGC::process removing MyZone.rgw.buckets.data:c5d78a84-c400-4c18-a0be-5d12df626549.4659.5__shadow_multipart_foo.2~OxemP2sKXW07N2--FqAxVV5Yg72boh5.4_1 2026-03-20T18:08:19.978 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.979+0000 7f40a2730900 5 garbage collection: RGWGC::process removing entries, marker: 2026-03-20T18:08:19.980 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.981+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2026-03-20T18:08:19.981 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.982+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.982 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.982+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.982 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.982+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.983 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.984+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2026-03-20T18:08:19.984 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.985+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.985 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.985+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.985 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.985+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.986 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.986+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2026-03-20T18:08:19.987 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.987+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.987 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.988+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.987 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.988+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.988 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.989+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2026-03-20T18:08:19.990 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.991+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.990 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.991+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.990 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.991+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.991 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.992+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2026-03-20T18:08:19.992 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.993+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.993 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.994+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.993 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.994+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.994 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.995+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2026-03-20T18:08:19.996 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.997+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.996 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.997+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.996 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.997+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.997 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.997+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2026-03-20T18:08:19.998 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.998+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.998 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.999+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:19.998 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:19.999+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:19.999 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.000+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2026-03-20T18:08:20.000 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.001+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:20.001 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.001+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:20.001 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.001+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:20.002 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.002+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2026-03-20T18:08:20.003 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.003+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:20.003 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.004+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:20.003 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.004+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:20.004 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.005+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2026-03-20T18:08:20.006 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.006+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:20.006 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.007+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:20.006 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.007+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:20.007 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.007+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2026-03-20T18:08:20.008 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.009+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:20.009 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.009+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:20.009 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.010+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:20.010 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.011+0000 7f40a2730900 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2026-03-20T18:08:20.011 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.012+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:20.011 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.012+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2026-03-20T18:08:20.012 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.012+0000 7f40a2730900 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2026-03-20T18:08:20.016 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.017+0000 7f40a2730900 20 remove_watcher() i=0 2026-03-20T18:08:20.016 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.017+0000 7f40a2730900 2 removed watcher, disabling cache 2026-03-20T18:08:20.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.017+0000 7f40a2730900 20 remove_watcher() i=7 2026-03-20T18:08:20.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.017+0000 7f40a2730900 20 remove_watcher() i=2 2026-03-20T18:08:20.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.017+0000 7f40a2730900 20 remove_watcher() i=4 2026-03-20T18:08:20.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.017+0000 7f40a2730900 20 remove_watcher() i=3 2026-03-20T18:08:20.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.017+0000 7f40a2730900 20 remove_watcher() i=6 2026-03-20T18:08:20.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.017+0000 7f40a2730900 20 remove_watcher() i=5 2026-03-20T18:08:20.017 INFO:teuthology.orchestra.run.vm01.stderr:2026-03-20T18:08:20.018+0000 7f40a2730900 20 remove_watcher() i=1 2026-03-20T18:08:20.022 DEBUG:tasks.rgw.client.1:waiting for process to exit 2026-03-20T18:08:20.022 INFO:teuthology.orchestra.run:waiting for 300 2026-03-20T18:08:20.027 INFO:tasks.rgw.client.1.vm03.stdout:2026-03-20T18:08:20.026+0000 1183c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/ceph.client.1.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends beast port=80 -n client.1 --cluster ceph -k /etc/ceph/ceph.client.1.keyring --log-file /var/log/ceph/rgw.ceph.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.1.sock --foreground (PID: 61572) UID: 0 2026-03-20T18:08:20.030 INFO:tasks.rgw.client.1.vm03.stdout:2026-03-20T18:08:20.028+0000 bf16f40 -1 shutting down 2026-03-20T18:08:23.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2026-03-20T18:08:43.480 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2026-03-20T18:08:43.481 DEBUG:teuthology.exit:Finished running handlers